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Why  GAO  Did  This  Study 

The  Department  of  Defense  (DOD) 
has  long  been  challenged  in 
implementing  key  information 
technology  (IT)  management 
controls  on  its  thousands  of 
business  system  investments.  For 
this  and  other  reasons,  GAO  has 
designated  DOD’s  business  systems 
modernization  efforts  as  high-risk. 
One  of  the  larger  business  system 
investments  is  the  Department  of 
the  Navy’s  Enterprise  Resource 
Planning  (Navy  ERP)  program. 
Initiated  in  2003,  the  program  is  to 
standardize  the  Navy’s  business 
processes,  such  as  acquisition  and 
financial  management.  It  is  being 
delivered  in  increments,  the  first  of 
which  is  to  cost  about  $2.4  billion 
over  its  useful  life  and  be  fully 
deployed  in  fiscal  year  2013.  GAO 
was  asked  to  determine  whether 
key  IT  management  controls  are 
being  implemented  on  the  program. 
To  do  this,  GAO  analyzed,  for 
example,  requirements 
management,  economic 
justification,  earned  value 
management,  and  risk 
management. 


What  GAO  Recommends 


GAO  is  making  recommendations 
to  the  Secretary  of  Defense  aimed 
at  improving  cost  and  schedule 
estimating,  earned  value 
management,  and  risk  management 
weaknesses.  DOD  largely  agreed 
with  GAO’s  recommendations  and 
described  actions  planned  or  under 
way  to  address  them. 


To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  GAO-08-896. 

For  more  information,  contact  Randolph  C. 
Hite  at  (202)  512-3439  or  hiter@gao.gov. 


What  GAO  Found 

DOD  has  implemented  key  IT  management  controls  on  its  Navy  ERP  program 
to  varying  degrees  of  effectiveness.  To  its  credit,  the  control  associated  with 
managing  system  requirements  is  being  effectively  implemented.  In  addition, 
important  aspects  of  other  controls  have  at  least  been  partially  implemented, 
including  those  associated  with  economically  justifying  investment  in  the 
program  and  proactively  managing  program  risks.  Nevertheless,  other  aspects 
of  these  controls,  as  well  as  the  bulk  of  what  is  needed  to  effectively 
implement  earned  value  management,  which  is  a  recognized  means  for 
measuring  program  progress,  have  not  been  effectively  implemented.  Among 
other  things,  these  control  weaknesses  have  contributed  to  the  more  than  2- 
year  schedule  delay  and  the  almost  $600  million  cost  overruns  already 
experienced  on  the  program  since  it  began,  and  they  will  likely  contribute  to 
future  delays  and  overruns  if  they  are  not  corrected.  Examples  of  the 
weaknesses  are  provided  below. 

•  Investment  in  the  program  has  been  economically  justified  on  the  basis  of 
expected  benefits  that  far  exceed  estimated  costs  ($8.6  billion  versus  $2.4 
billion  over  a  20-year  life  cycle).  However,  important  estimating  practices, 
such  as  using  historical  cost  data  from  comparable  programs  and  basing 
the  cost  estimate  on  a  reliable  schedule  baseline  were  not  employed. 

While  these  weaknesses  are  important  because  they  limit  the  reliability  of 
the  estimates,  GAO’s  analysis  shows  that  they  would  not  have  altered  the 
estimates  to  the  point  of  not  producing  a  positive  return  on  investment. 

•  Earned  value  management  has  not  been  effectively  implemented.  To  its 
credit,  the  program  office  has  elected  to  implement  program-level  earned 
value  management.  In  doing  so,  however,  basic  prerequisites  for 
effectively  managing  earned  value  have  not  been  executed.  In  particular, 
the  integrated  master  schedule  was  not  derived  in  accordance  with  key 
estimating  practices,  and  an  integrated  baseline  review  has  not  been 
performed  on  any  of  the  first  increment’s  releases. 

•  A  defined  process  for  proactively  avoiding  problems,  referred  to  as  risk 
management,  has  been  established,  but  risk  mitigation  strategies  have  not 
been  effectively  implemented  for  all  significant  risks,  such  as  those 
associated  with  data  conversion  and  organizational  change  management, 
as  well  the  risks  associated  with  the  above-cited  weaknesses. 

The  reasons  that  program  management  and  oversight  officials  cited  for  these 
practices  not  being  executed  range  from  the  complexity  and  challenges  of 
managing  and  implementing  a  program  of  this  size  to  limitations  in  the 
program  office’s  scope  and  authority.  Notwithstanding  the  effectiveness  with 
which  important  aspects  of  several  controls  have  been  implemented,  the 
above-cited  weaknesses  put  DOD  at  risk  of  investing  in  a  system  solution  that 
does  not  optimally  support  corporate  mission  needs  and  mission 
performance,  and  meet  schedule  and  cost  commitments. 
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Abbreviations 

BEA 

business  enterprise  architecture 

BTA 

Business  Transformation  Agency 

CIO 

Chief  Information  Officer 

DBSMC 

Defense  Business  Systems  Management  Committee 

DOD 

Department  of  Defense 

DON 

Department  of  the  Navy 

EVM 

earned  value  management 

FOC 

full  operational  capability 

GCSS-MC 

Global  Combat  Support  System-Marine  Corps 

IOC 

initial  operational  capability 

IRB 

investment  review  board 

IT 

information  technology 

MDA 

milestone  decision  authority 

NAVAIR 

Naval  Air  Systems  Command 

NAVSEA 

Naval  Sea  Systems  Command 

NAVSUP 

Naval  Supply  Systems  Command 

Navy  ERP 

Navy  Enterprise  Resource  Planning 

NTCSS 

Naval  Tactical  Command  Support  System 

OMB 

Office  of  Management  and  Budget 

PMB 

performance  measurement  baseline 

SAP 

Systems  Applications  and  Products 

SPAWAR 

Space  and  Naval  Warfare  Systems  Command 

TC-AIMS  II 

Transportation  Coordinators’  Automated  Information  for 
Movements  System 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


September  8,  2008 

The  Honorable  Daniel  K.  Akaka 
Chairman 

The  Honorable  John  Thune 
Ranking  Member 

Subcommittee  on  Readiness  and  Management  Support 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  John  Ensign 
United  States  Senate 

For  decades,  the  Department  of  Defense  (DOD)  has  been  challenged  in 
modernizing  its  timeworn  business  systems.1  In  1995,  we  designated  DOD’s 
business  systems  modernization  program  as  high  risk,  and  continue  to  do 
so  today.2  Our  reasons  include  the  modernization’s  large  size,  complexity, 
and  its  critical  role  in  addressing  other  long-standing  transformation  and 
financial  management  challenges.  Other  reasons  are  that  DOD  has  yet  to 
institutionalize  key  system  modernization  management  controls,  and  it  has 
not  demonstrated  the  ability  to  consistently  deliver  promised  system 
capabilities  and  benefits  on  time  and  within  budget. 

Nevertheless,  DOD  continues  to  invest  billions  of  dollars  in  thousands  of 
business  systems,  including  about  a  hundred  that  the  department  has 
labeled  as  business  transformational  programs,  12  of  which  account  for 
about  50  percent  of  these  programs’  costs.  The  Navy  Enterprise  Resource 
Planning  (Navy  ERP)  program  is  one  such  program.  Initiated  in  2003,  Navy 
ERP  is  to  standardize  the  Navy’s  acquisition,  financial,  program 
management,  maintenance,  plant  and  wholesale  supply,  and  workforce 
management  business  processes  across  its  dispersed  organizational 
environment.  As  envisioned,  the  program  consists  of  a  series  of  major 
increments,  the  first  of  which  includes  three  releases  and  is  expected  to 
cost  approximately  $2.4  billion  over  its  20-year  life  cycle  and  to  be  fully 
operational  in  fiscal  year  2013. 


business  systems  are  information  systems,  including  financial  and  nonfinancial  systems, 
that  support  DOD  business  operations,  such  as  civilian  personnel,  finance,  health,  logistics, 
military  personnel,  procurement,  and  transportation. 

2GAO,  High-Risk  Series:  An  Update,  GAO-07-310  (Washington,  D.C.:  January  2007). 


Page  1 


GAO-08-896  DOD  Business  Systems  Modernization 


As  agreed,  our  objective  was  to  determine  whether  the  Department  of  the 
Navy  (DON)3  is  effectively  implementing  information  technology  (IT) 
management  controls  on  Navy  ERP.  To  accomplish  this,  we  focused  on 
the  program’s  first  increment  by  analyzing  a  range  of  program 
documentation  and  interviewing  cognizant  officials  relative  to  the 
following  management  areas:  architectural  alignment,  economic 
justification,  earned  value  management  (EVM),  requirements  management, 
and  risk  management.  We  conducted  this  performance  audit  from  June 
2007  to  September  2008,  in  accordance  with  generally  accepted 
government  auditing  standards.  Those  standards  require  that  we  plan  and 
perform  the  audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a 
reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit 
objective.  We  believe  that  the  evidence  obtained  provides  a  reasonable 
basis  for  our  findings  and  conclusions  based  on  our  audit  objective. 
Additional  details  on  our  objective,  scope,  and  methodology  are  in 
appendix  I. 


Results  in  Brief 


DOD  has  implemented  key  IT  management  controls  on  its  Navy  ERP 
program  to  varying  degrees  of  effectiveness.  To  its  credit,  one  of  the 
controls  has  been  fully  implemented;  important  aspects  of  other  controls 
have  not.  Collectively,  these  management  controls  are  to  ensure  that  a 
given  system  investment  represents  the  right  solution  to  filling  a  mission 
need,  meaning  that  the  system  is  defined  to  (1)  minimize  overlap  and 
duplication  and  maximize  interoperability  with  related  systems  and 
(2)  produce  mission  benefits  commensurate  with  costs  over  its  useful  life. 
The  controls  are  also  to  ensure  that  the  system  is  acquired  and  deployed 
the  right  way,  meaning  that  it  is  done  in  a  way  to  maximize  the  chances  of 
delivering  defined  system  capabilities  and  benefits  on  time  and  within 
budget.  Given  that  deployment  of  Navy  ERP  is  more  than  2  years  behind 
schedule  and  is  to  cost  about  $570  million4  more  than  was  originally 
envisioned,  these  goals  have  already  not  been  fully  met,  in  part  because 
DOD  program  management  and  oversight  entities  have  not  fully 
implemented  several  key  IT  management  controls.  As  a  result,  the 
department  has  yet  to  adequately  demonstrate  that  the  program’s  first 
increment,  as  it  has  been  defined,  is  the  right  solution,  and  it  is  likely  that 


3The  Department  of  the  Navy  is  a  major  component  of  DOD,  consisting  of  two  uniformed 
services:  the  Navy  and  the  Marine  Corps. 

4This  amount  is  the  difference  between  the  September  2007  estimated  life  cycle  cost  of 
$2,445.0  million  and  the  September  2003  estimated  life  cycle  cost  of  $1,873.1  million. 
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the  department  will  continue  to  add  to  the  program’s  cost  overruns  and 
schedule  delays  that  the  program  has  already  experienced  to  date.  The 
strengths  and  weaknesses  associated  with  each  of  the  IT  management 
controls  that  we  evaluated  are  described  here: 

•  Navy  ERP  compliance  with  DOD’s  federated  business  enterprise 
architecture  (BEA)  has  not  been  sufficiently  demonstrated.  To  its  credit, 
the  program  office  followed  DOD’s  architecture  compliance  assessment 
guidance  and  used  the  related  assessment  tool.  However,  this  guidance 
and  tool  do  not  adequately  provide  for  addressing  all  relevant  aspects  of 
architectural  compliance.  Specifically,  the  program’s  compliance 
assessment  (1)  did  not  include  all  relevant  architecture  products,  such  as 
those  that  describe  technical  and  system  elements;  (2)  was  not  used  to 
identify  potential  areas  of  duplication  across  programs;  and  (3)  did  not 
address  compliance  with  the  DON  enterprise  architecture.  These 
important  steps  were  not  performed  because  of  policy,  guidance,  and  tool 
limitations,  and  because  aspects  of  the  corporate  BEA  and  the  DON 
enterprise  architecture,  which  are  both  major  components  of  DOD’s 
federated  BEA,  have  yet  to  be  sufficiently  defined  to  permit  thorough 
compliance  determinations  in  these  areas.  In  addition,  program  oversight 
and  approval  authorities  did  not  validate  the  program  office’s  compliance 
assessments.  As  a  result,  the  department  does  not  have  a  sufficient  basis 
for  knowing  if  Navy  ERP  has  been  defined  to  minimize  overlap  with,  and 
duplication  of  other  programs’  functions,  and  is  being  defined  and 
designed  to  maximize  interoperability  among  related  programs. 

•  Investment  in  Navy  ERP  has  been  economically  justified  on  the  basis  of 
expected  life  cycle  benefits  that  far  exceed  estimated  life  cycle  costs 
($8.6  billion  versus  $2.4  billion  over  a  20-year  life  cycle).  However,  these 
benefit  estimates  have  not  been  subject  to  analysis  of  how  uncertainty  in 
assumptions  and  data  could  impact  them,  as  prescribed  in  relevant 
guidance.  According  to  program  officials,  such  uncertainty  analysis  is  not 
warranted  because  they  have  taken  and  continue  to  take  steps  to  validate 
the  assumptions  and  the  data,  such  as  using  the  latest  budget  data 
associated  with  retiring  legacy  systems,  and  monitoring  changes  to  the 
systems’  retirement  dates.  While  these  steps  are  positive,  they  do  not 
eliminate  the  need  for  uncertainty  analysis.  Accordingly,  we  assessed  key 
uncertainty  variables  and  found  that  while  the  inherent  uncertainty  in  the 
estimates  would  reduce  expected  savings,  the  reduction  would  be  small 
relative  to  a  total  benefit  estimate  of  $8.6  billion. 

With  respect  to  the  cost  estimate,  our  analysis  showed  that  it  was  not 
derived  using  all  key  estimating  practices  contained  in  relevant  guidance. 
For  example,  the  estimate  was  not  grounded  in  a  historical  record  of 
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comparable  data  from  similar  programs  and  was  not  based  on  a  reliable 
schedule  baseline,  which  are  both  necessary  to  having  a  cost  estimate  that 
can  be  considered  credible  and  accurate.  These  practices  were  not 
employed  for  various  reasons,  including  DOD’s  lack  of  historical  data  from 
similar  programs  and  the  lack  of  an  integrated  master  schedule  for  the 
program’s  first  increment  that  includes  all  three  releases.  Notwithstanding 
the  fact  that  these  limitations  could  materially  increase  the  $2.4  billion 
cost  estimate,  it  is  nevertheless  unlikely  that  they  would  increase  the  cost 
estimate  to  a  level  close  to  the  uncertainty-adjusted  benefit  expectations. 
Therefore,  we  have  no  reason  to  believe  that  Navy  ERP  will  not  produce  a 
positive  return  on  investment. 

•  EVM,  which  is  a  means  for  determining  and  disclosing  actual  program 
performance  in  comparison  with  estimates,  is  not  being  effectively 
implemented  in  Navy  ERP.  To  its  credit,  the  program  office  has  elected  to 
implement  program-level  EVM.6  In  doing  so,  however,  basic  EVM  activities 
have  not  been  executed,  which  has  produced,  and  will  likely  continue  to 
produce,  actual  program  costs  and  schedules  that  do  not  track  close  to 
estimates.  For  example,  an  integrated  baseline  review,  which  is  to  verify 
that  the  program’s  costs  and  schedule  are  reasonable  given  the  program’s 
scope  of  work  and  associated  risks,  has  not  been  performed  on  any  of  the 
first  increment’s  releases.  According  to  program  officials,  this  is  because 
of  the  time  it  took  to  establish  program-level  EVM  capabilities  and  because 
of  their  focus  on  deploying  and  stabilizing  the  first  release.  However,  they 
recently  stated  that  one  has  been  tentatively  scheduled  for  August  2008.  By 
not  having  an  integrated  master  schedule  that  has  been  subject  to  a 
baseline  review,  as  well  as  not  employing  other  industry  standards  as 
discussed  in  this  report,  Navy  ERP  will  be  challenged  in  implementing 
EVM  effectively,  and  cost  overruns  and  lengthy  schedule  delays  beyond 
those  already  experienced  by  the  program  will  likely  occur.  Our  analysis 
of  the  latest  estimate  for  completing  just  the  budgeted  development  work 
for  all  three  releases,  which  is  about  $844  million,  shows  that  this  estimate 
will  most  likely  have  an  overrun  of  about  $152  million. 

•  An  important  requirements  management  activity  has  been  effectively 
implemented  in  Navy  ERP.  Specifically,  the  program  office  has  ensured 
that  system  requirements  for  the  first  release  are  traceable,  as  evidenced 
by  our  analysis  of  60  randomly  selected  system-level  requirements  in 


6Program-level  means  that  EVM  covers  all  aspects  of  the  program,  regardless  of  whether 
they  are  performed  by  the  government  or  the  contractor.  In  contrast,  EVM  has  historically 
been  implemented  on  a  contract-by-contract  basis,  and  has  not  included  government 
performed  work. 
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which  we  confirmed  that  58  are  traceable  backward  to  operational 
requirements  and  forward  to  design  specifications  and  test  results.  Such 
traceability  is  important  because  it  increases  the  chances  of  delivering  a 
system  that  performs  as  intended.  Our  analysis  of  requirements  in  this 
sample  also  confirmed  that  system  requirements  that  had  been  reallocated 
from  the  first  release  to  the  other  releases  were  traceable,  thus 
demonstrating  traceability  among  product  releases. 

•  The  program  office  has  defined  a  process  for  proactively  managing  risks 
that  reflects  key  practices  governing  how  this  IT  management  control 
should  be  performed.  However,  it  has  not  effectively  implemented  this 
process  for  all  identified  risks.  In  particular,  steps  taken  to  mitigate  the 
risks  associated  with  converting  data  from  existing  systems  to  the  new 
system  and  positioning  user  organizations  for  the  operational  changes 
associated  with  the  new  system  were  not  effective.  According  to  program 
officials,  these  mitigation  strategies  could  not  be  effectively  implemented 
because  the  program  office  does  not  have  the  authority  to  compel  the  user 
organizations  to  execute  their  part  of  the  mitigation  strategy.  As  a  result, 
the  first  user  organization  to  receive  the  Navy  ERP  experienced  significant 
problems  during  recent  operational  testing,  and  these  problems  will  take 
additional  time  and  resources  to  correct.  In  addition,  not  all  known  risks 
have  been  captured  in  the  risk  inventory.  For  example,  the  risks 
associated  with  the  two  above  discussed  control  weaknesses  (not  having 
adequately  demonstrated  the  program’s  architectural  alignment  and  not 
having  implemented  program-level  EVM  according  to  industry  practices) 
are  not  included  in  the  risk  inventory  and  are  thus  not  being  disclosed  and 
addressed.  This  is  important  because  not  having  effectively  addressed 
such  risks  has  contributed  to  schedule  delays  and  will  likely  contribute  to 
more  cost  and  schedule  shortfalls. 

Notwithstanding  the  effectiveness  with  which  several  program 
management  controls  have  been  implemented  on  Navy  ERP,  the  above- 
cited  weaknesses  in  implementing  other  management  controls  put  DOD  at 
risk  of  investing  in  a  system  solution  that  does  not  optimally  support 
corporate  mission  needs  and  mission  performance  and  continuing  to  fall 
short  of  program  schedule  and  cost  commitments.  Accordingly,  we  are 
making  recommendations  to  the  Secretary  of  Defense  aimed  at  addressing 
the  cost  and  schedule  estimating,  EVM,  and  risk  management  weaknesses, 
thereby  better  ensuring  that  the  program  is  managed  to  deliver  the  right 
solution,  the  right  way.  We  are  not  making  recommendations  in  this  report 
for  addressing  the  architecture  compliance  weaknesses  because  we 
recently  completed  other  work  that  is  more  broadly  focused  on  this 
management  control  across  multiple  programs. 
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In  written  comments  on  a  draft  of  this  report,  signed  by  the  Deputy  Under 
Secretary  of  Defense  (Business  Transformation)  and  reprinted  in  appendix 
II,  the  department  stated  that  it  concurred  with  two,  and  partially 
concurred  with  two,  of  our  four  recommendations.  Further,  it  stated  that  it 
has  taken  steps  to  address  some  of  our  recommendations,  adding  that  it  is 
committed  to  implementing  recommendations  that  contribute  to  the 
program’s  success. 

In  partially  concurring  with  two  of  the  recommendations,  DOD  concurred 
with  most  aspects  of  both.  Nevertheless,  for  our  recommendation  aimed  at 
improving  the  program’s  cost  estimates,  it  stated  that  it  did  not  concur 
with  one  aspect — ensuring  that  future  cost  estimates  reflect  the  risk  of  not 
having  cost  data  for  comparable  programs.  In  doing  so,  DOD  stated  that 
while  the  program  had  limited  comparable  data  on  which  to  base  the 
program’s  cost  estimate,  it  had  accounted  for  this  limitation  in  the  cost 
estimate’s  uncertainty  analysis.  We  do  not  agree  that  this  risk  was 
reflected  in  the  uncertainty  analysis.  We  examined  this  analysis  as  part  of 
our  review  and  found  that  it  did  not  recognize  this  risk.  Moreover,  DOD’s 
comments  did  not  include  any  evidence  to  the  contrary. 

In  addition,  for  our  recommendation  aimed  at  improving  the  program’s 
integrated  master  schedule,  the  department  partially  concurred  with  one 
of  the  five  components  of  the  recommendation — defining  a  critical  path 
that  incorporates  all  three  releases  of  the  system.  In  doing  so,  DOD  stated 
what  our  report  already  recognized,  namely  that  the  schedule  reflects  a 
separate  critical  path  for  each  release  and  that  this  is  due  to  the  size  and 
complexity  of  the  releases.  However,  DOD  offers  no  new  information  in  its 
comments.  Further,  our  report  also  recognizes  that  to  be  successful,  large 
and  complex  programs  that  involve  thousands  of  activities  need  to  ensure 
that  their  schedules  integrate  these  activities.  In  this  regard,  we  support 
the  department’s  commitment  to  explore  the  feasibility  of  implementing 
this  part  of  our  recommendation. 

While  concurring  with  all  components  of  our  other  two  recommendations, 
the  department  nevertheless  provided  comments  relative  to  the  program’s 
use  of  EVM  to  explain  why  Release  1.0  of  the  system  was  not  subject  to  an 
integrated  baseline  review.  For  several  reasons  discussed  in  the  agency 
comments  section  of  this  report,  we  do  not  agree  with  these  additional 
comments.  In  addition,  the  department’s  comments  described  actions 
planned  or  under  way  to  address  our  recommendations.  If  fully  and 
properly  implemented,  DOD’s  described  actions  should  go  a  long  way  in 
addressing  the  weaknesses  that  we  identify  in  the  report. 
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Background 


DON’s  primary  mission  is  to  organize,  train,  maintain,  and  equip  combat- 
ready  naval  forces  capable  of  winning  the  global  war  on  terrorism  and  any 
other  armed  conflict,  deterring  aggression  by  would-be  foes,  preserving 
freedom  of  the  seas,  and  promoting  peace  and  security.  To  support  this 
mission,  DON  performs  a  variety  of  interrelated  and  interdependent 
business  functions  (e.g.,  acquisition  and  financial  management),  relying 
heavily  on  IT  systems.  In  fiscal  year  2008,  DON’s  budget  for  business 
systems  and  associated  infrastructure  was  about  $2.7  billion,  of  which  $2.2 
billion  was  allocated  to  operations  and  maintenance  of  existing  systems 
and  the  remaining  $500  million  to  systems  in  development  and 
modernization.  Of  the  approximately  3,000  business  systems  that  DOD 
reports  in  its  current  inventory,  DON  accounts  for  904,  or  about  30 
percent,  of  the  total.  Navy  ERP  is  one  such  system  investment. 


Navy  ERP:  A  Brief  In  July  2003,  the  Assistant  Secretary  of  the  Navy  for  Research, 

Description  Development,  and  Acquisition  established  Navy  ERP  to  “converge”  four 

separate  pilot  programs  that  were  under  way  at  four  separate  Navy 
commands.6  This  program  is  to  leverage  a  commercial,  off-the-shelf 
software  known  as  an  enterprise  resource  planning  product.  Such 
products  consist  of  multiple,  integrated  functional  modules  that  perform  a 
variety  of  business-related  tasks,  such  as  acquisition  and  financial 
management.  Table  1  provides  a  brief  description  and  status  of  each  of  the 
pilots. 


6These  commands  are  Naval  Air,  Naval  Sea,  Space  and  Naval  Warfare,  and  Naval  Supply 
Systems. 
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Table  1:  Description  and  Status  of  the  Navy  ERP  Pilots 

Pilot 

Description  and  status 

SIGMA 

Deployed  at  Naval  Air  Systems  Command  (NAVAIR)3  to  support  and  link 
such  business  functions  as  program  management,  contracting,  financial, 
and  human  resources  management.  It  was  retired  in  the  first  quarter  of 
fiscal  year  2008. 

CABRILLO 

Deployed  at  Space  and  Naval  Warfare  Systems  Command  (SPAWAR)b  to 
support  Navy  Working  Capital  Fund  financial  management.  It  is  to  be 
retired  in  fiscal  year  2010. 

NEMAIS 

Deployed  at  Naval  Sea  Systems  Command  (NAVSEA)0  to  support 
regional  maintenance,  including  intermediate-level  maintenance" 
management  and  human  resources.  It  is  to  be  retired  in  fiscal  year  201 1 . 

SMART 

Deployed  at  Naval  Supply  Systems  Command  (NAVSUP)8  and  NAVAIR  to 
support  national  and  local  supply  management,  intermediate-level 
maintenance  management  and  to  interface  with  aviation  depots.  It  was 
retired  in  fiscal  year  2005. 

Source:  GAO  analysis  of  DON  data. 


“NAVAIR  is  responsible  for  developing,  delivering,  and  supporting  aircraft  and  weapons  used  by 
sailors  and  marines. 

bSPAWAR  is  responsible  for  developing,  delivering,  and  supporting  specialized  command  and  control 
technologies,  business  information  technology,  and  space  capabilities. 

"NAVSEA  is  responsible  for  acquiring  and  maintaining  the  Navy's  ships  and  submarines. 

"Intermediate-level  maintenance  is  for  repair  or  maintenance  of  items  that  do  not  have  to  go  to  the 
depot  level  for  major  work  but  cannot  be  maintained  or  repaired  at  the  organizational  level. 

“NAVSUP  is  responsible  for  supply,  fuel,  and  transportation,  as  well  as  other  logistics  programs. 


According  to  DOD,  Navy  ERP  is  to  address  the  Navy’s  long-standing 
problems  related  to  financial  transparency  and  asset  visibility.  Specifically, 
the  program  is  intended  to  standardize  the  Navy’s  acquisition,  financial, 
program  management,  maintenance,  plant  and  wholesale  supply,  and 
workforce  management  business  processes  across  its  dispersed 
organizational  components.  When  the  program  is  fully  implemented,  it  is 
to  support  over  86,000  users. 

Navy  ERP  is  being  developed  in  a  series  of  increments  using  the  Systems 
Applications  and  Products  (SAP)  commercial  software  package, 
augmented  as  needed  by  customized  software.  SAP  consists  of  multiple, 
integrated  functional  modules  that  perform  a  variety  of  business  related 
tasks,  such  as  finance  and  acquisition.  The  first  increment,  called 
Template  1,  is  currently  the  only  funded  portion  of  the  program  and 
consists  of  three  releases:  1.0  Financial  and  Acquisition,  1.1  Wholesale  and 
Retail  Supply,  and  1.2  Intermediate-Level  Maintenance.  Release  1.0  is  the 
largest  of  the  three  releases  in  terms  of  the  functional  requirements  being 
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addressed.  Specifically,  it  is  to  provide  about  56  percent  of  Template  1 
requirements.  See  table  2  for  a  description  of  these  releases. 


Table  2:  Navy  ERP  Template  1  Releases 

Release 

Functionality 

1.0  Financial  and  Acquisition 

General  Fund  and  Navy  Working  Capital  Fund 
finance  applications,  such  as  billing,  budgeting,  and 
cost  planning. 

Acquisition  applications,  such  as  activity  based 
costing,  contract  awards,  and  budget  exhibits. 

Workforce  management  applications,  such  as 
personnel  administration,  and  training  and  events 
management. 

1.1  Wholesale  and  Retail  Supply 

Wholesale  applications,  such  as  supply  and  demand 
planning,  order  fulfillment,  and  supply  forecasting. 

Retail  supply  applications,  such  as  inventory 
management,  supply  and  demand  processing,  and 
warehouse  management. 

1.2  Intermediate-Level 

Maintenance  applications,  such  as  maintenance 

Maintenance 

management,  quality  management,  and  calibration 
management. 

Source:  GAO  analysis  of  DON  data. 


DON  estimates  the  life  cycle  cost  for  the  program’s  first  increment  to  be 
about  $2.4  billion,  including  about  $1  billion  for  acquisition,  and  $1.4 
billion  for  operations  and  maintenance.  The  life  cycle  cost  of  the  entire 
program  has  not  yet  been  determined  because  future  increments  have  not 
been  defined.  The  program  office  reported  that  approximately  $400  million 
was  spent  from  fiscal  year  2004  through  fiscal  year  2007  on  the  first 
increment.  For  fiscal  year  2008,  about  $200  million  is  planned  to  be  spent. 


Program  Oversight  and 
Management  Roles  and 
Responsibilities 


To  manage  the  acquisition  and  deployment  of  Navy  ERP,  DON  established 
a  program  management  office  within  the  Program  Executive  Office  for 
Executive  Information  Systems.  The  program  office  manages  the 
program’s  scope  and  funding  and  is  responsible  for  ensuring  that  the 
program  meets  its  objectives.  To  accomplish  this,  the  program  office  is 
responsible  for  key  program  management  areas,  such  as  architectural 
alignment,  economic  justification,  earned  value  management, 
requirements  management,  and  risk  management.  In  addition,  various 
DOD  and  DON  organizations  share  program  oversight  and  review 
activities.  A  listing  of  key  entities  and  their  roles  and  responsibilities  is  in 
table  3. 
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Table  3:  Organizations  Responsible  for  Navy  ERP  Oversight  and  Management 

Entity 

Roles  and  responsibilities 

Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics 

Serves  as  the  milestone  decision  authority  (MDA),  which  according  to  DOD,  has 
overall  responsibility  for  the  program,  to  include  approving  the  program  to  proceed 
through  its  acquisition  cycle  on  the  basis  of,  for  example,  the  acquisition  plan,  an 
independently  evaluated  economic  analysis,  and  the  Acquisition  Program  Baseline. 

Assistant  Secretary  of  the  Navy,  Research, 
Development,  and  Acquisition 

Serves  as  DON’S  oversight  organization  for  the  program,  to  include  enforcement  of 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  policies  and 
procedures. 

Department  of  the  Navy,  Program  Executive 
Office  for  Executive  Information  Systems 

Oversees  a  portfolio  of  large-scale  projects  and  programs  designed  to  enable  common 
business  processes  and  provide  standard  capabilities,  to  include  reviewing  the 
acquisition  plan,  economic  analysis,  and  the  Acquisition  Program  Baseline  prior  to 
approval  by  the  MDA. 

Department  of  the  Navy  Chief  Information 
Officer  (CIO) 

Supports  the  department’s  planning,  programming,  budgeting,  and  execution 
processes  by  ensuring  that  the  program  has  achievable  and  executable  goals  and 
conforms  to  financial  management  regulations,  and  DON,  DOD,  and  federal  IT  policies 
in  several  areas  (e.g.,  security,  architecture,  and  investment  management);  works 
closely  with  the  program  office  during  milestone  review  assessments. 

Office  of  the  Secretary  of  Defense,  Office  of 
the  Director  for  Program  Analysis  and 
Evaluation 

Verifies  and  validates  the  reliability  of  cost  and  benefit  estimates  found  in  economic 
analyses  and  provides  its  results  to  the  MDA. 

Naval  Center  for  Cost  Analyses 

Performs  independent  costs  estimates. 

Defense  Business  Systems  Management 
Committee  (DBSMC) 

Serves  as  the  highest  ranking  governance  body  for  business  systems  modernization 
activities  and  approves  investments  costing  more  than  $1  million,  as,  for  example, 
being  compliant  with  the  BEA. 

Investment  Review  Board  (IRB) 

Reviews  business  system  investments  and  has  responsibility  for  recommending 
certification  for  all  business  system  investments  costing  more  than  $1  million  that  are 
asserted  as  compliant  with  the  BEA. 

Business  Transformation  Agency  (BTA) 

Coordinates  business  transformation  efforts  across  DOD  and  supports  the  IRBs  and 
DBSMC. 

Navy  ERP  Program  Management  Office 

Performs  day-to-day  program  management  and,  as  such,  is  the  single  point  of 
accountability  for  managing  the  program’s  objectives  through  development, 
deployment,  and  sustainment. 

Source:  DOD. 

Overview  of  Navy  ERP’s 
Status 

The  first  increment  of  Navy  ERP  is  currently  in  the  production  and 
deployment  phase  of  the  defense  acquisition  system.7  The  system  consists 

of  five  key  program  life  cycle  phases  and  three  related  milestone  decision 
points.  These  five  phases  and  related  milestones,  along  with  a  summary  of 


The  defense  acquisition  system  is  a  framework-based  approach  that  is  intended  to 
translate  mission  needs  and  requirements  into  stable,  affordable,  and  well-managed 
acquisition  programs. 
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key  program  activities  completed  during  or  planned  for  each  phase,  are  as 
follows: 


1.  Concept  Refinement:  The  purpose  of  this  phase  is  to  refine  the  initial 
system  solution  (concept)  and  create  a  strategy  for  acquiring  the 
solution.  This  phase  began  in  July  2003,  at  which  time  DON  began  to 
converge  the  four  pilot  programs  into  Navy  ERP  and  developed  its  first 
cost  estimate  in  September  2003.  This  phase  of  the  program  was 
combined  with  the  next  phase,  thus  creating  a  combined  Milestone  A/B 
decision  point. 

2.  Technology  Development:  The  purpose  of  this  phase  is  to  determine 
the  appropriate  set  of  technologies  to  be  integrated  into  the  investment 
solution  by  iteratively  assessing  the  viability  of  the  various 
technologies  while  simultaneously  refining  user  requirements.  During 
the  combined  Concept  Refinement  and  Technology  Development 
phase,  the  program  office  prepared  a  concept  of  operations  and 
operational  requirements  document;  performed  an  analysis  of 
alternatives,  business  case  analysis,  and  economic  analysis;  and 
established  its  first  Acquisition  Program  Baseline.  It  also  selected  SAP 
as  the  commercial  off-the-shelf  ERP  software.  The  combined  phase 
was  completed  in  August  2004,  when  the  MDA  approved  Milestone  A/B 
to  allow  the  program  to  move  to  the  next  phase. 

3.  System  Development  and  Demonstration:  The  purpose  of  this  phase  is 
to  develop  a  system  and  demonstrate  through  developer  testing  that 
the  system  can  function  in  its  target  environment.  This  phase  was 
completed  in  September  2007,  when  Release  1.0  passed  development 
testing  and  its  deployment  to  NAVAIR  began.  This  was  17  months  later 
than  the  program’s  original  schedule  set  in  August  2004  but  on  time 
according  to  the  revised  schedule  set  in  December  2006. 

In  September  2004,  the  program  office  awarded  a  $176  million  system 
integration  contract  to  BearingPoint  for  full  system  design, 
development,  and  delivery  using  SAP’s  off-the-shelf  product  and 
related  customized  software.  In  January  2006,  the  program  office  (1) 
reduced  the  contractor’s  scope  of  work  from  development  and 
integration  of  the  first  increment  to  only  development  of  the  first 
release  and  (2)  assumed  responsibility  and  accountability  for  overall 
system  integration.  According  to  the  program  office,  reasons  for  this 
change  included  the  need  to  change  the  development  plan  to  reflect 
improvements  in  the  latest  SAP  product  released  and  the  lack  of 
authority  by  the  contractor  to  adjudicate  and  reconcile  differences 
among  the  various  Navy  user  organizations  (i.e.,  Navy  commands). 
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In  December  2006,  the  program  office  revised  its  Acquisition  Program 
Baseline  to  reflect  an  increase  of  about  $461  million  in  the  life  cycle 
cost  estimate  due,  in  part,  to  restructuring  the  program  (e.g.,  changing 
the  order  of  the  releases,  changing  the  role  of  system  integrator  from 
contractor  to  the  program  office)  and  resolving  problems  related  to, 
among  other  things,  converting  data  from  legacy  systems  to  run  on 
Navy  ERP  and  establishing  interfaces  between  legacy  systems  and 
Navy  ERP.  In  addition,  the  program  office  awarded  a  $151  million 
contract  for  Release  1.1  and  1.2  configuration  and  development  to  IBM 
in  June  2007.  In  September  2007,  prior  to  entering  the  next  phase,  the 
program  revised  its  Acquisition  Program  Baseline  again  to  reflect  a  $9 
million  decrease  in  the  life  cycle  cost  estimate  and  a  5-month  increase 
in  its  program  schedule.  Soon  after,  the  MDA  approved  Milestone  C  to 
move  to  the  next  phase. 

4.  Production  and  Deployment:  The  purpose  of  this  phase  is  to  achieve 
an  operational  capability  that  satisfies  the  mission  needs,  as  verified 
through  independent  operational  test  and  evaluation,  and  to 
implement  the  system  at  all  applicable  locations.  This  phase  began  in 
September  2007,  focusing  first  on  achieving  initial  operational 
capability  (IOC)  of  Release  1.0  at  NAVAIR  by  May  2008.  This  date  is  22 
months  later  than  the  baseline  established  for  Milestone  A/B  in  August 
2004,  and  4  months  later  than  the  new  baseline  established  in 
September  2007.  According  to  program  documentation,  these  delays 
were  due,  in  part,  to  challenges  experienced  at  NAVAIR  in  converting 
data  from  legacy  systems  to  run  on  the  new  system  and  implementing 
new  business  procedures  associated  with  the  system. 

In  light  of  the  delays  at  NAVAIR  in  achieving  IOC,  the  deployment 
schedules  for  the  other  commands  were  also  revised.  Specifically, 
Release  1.0  is  still  to  be  deployed  at  NAVSUP  on  October  2008,  but 
Release  1.0  deployment  at  SPA  WAR  is  now  scheduled  18  months  later 
than  planned  (October  2009),  and  deployment  at  NAVSEA  general  fund 
and  Navy  Working  Capital  Fund  is  now  scheduled  to  be  12  months 
later  than  planned  (October  2010  and  2011,  respectively).  Because  of 
the  Release  1.0  delays,  Release  1.1  is  now  planned  for  deployment  at 
NAVSUP  7  months  later  than  planned  (February  2010).  Release  1.2  is 
still  scheduled  to  be  released  at  Regional  Maintenance  Centers  in 
October  2010. 
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The  program  office  is  currently  in  the  process  of  again  re-baselining 
the  program,  and  DON  plans  to  address  any  cost  overruns  through 
reprogramming  of  fiscal  year  2008  DON  funds.8  It  estimates  that  this 
phase  will  be  completed  with  full  operational  capability  (FOC)9  by 
August  2013  (26  months  later  than  the  baseline  established  in  2004,  and 
5  months  later  than  the  re-baseline  established  in  September  2007). 

5.  Operations  and  Support:  The  purpose  of  this  phase  is  to  operationally 
sustain  the  system  in  the  most  cost-effective  manner  over  its  life  cycle. 
In  this  phase,  the  program  plans  to  provide  centralized  support  to  its 
users  across  all  system  commands.  Each  deployment  site  is  expected 
to  perform  complementary  support  functions,  such  as  data 
maintenance. 

Overall,  Increment  1  was  originally  planned  to  reach  FOC  in  fiscal  year 
2011,  and  its  estimated  life  cycle  cost  was  about  $1.87  billion.10  The 
estimate  was  later  baselined11  in  August  2004  at  about  $2.0  billion.12  In 
December  2006  and  again  in  September  2007,  the  program  was  re¬ 
baselined.  FOC  is  now  planned  for  fiscal  year  2013,  and  the  estimated  life 
cycle  cost  is  about  $2.4  billion  (31  percent  increase  over  the  original 


Congressional  defense  committees  have  established  reprogramming  guidelines,  including 
setting  dollar  thresholds  that  direct  DOD  to  seek  the  prior  approval  of  the  committees 
before  executing  the  reprogramming  of  appropriated  funds.  In  accordance  with  these 
guidelines,  DOD’s  financial  management  regulation  provides  that  DOD  does  not  need 
congressional  committee  approval  when  the  amount  to  be  reprogrammed  falls  below 
certain  thresholds  (referred  to  as  a  "below-threshold  reprogramming").  As  relevant  here,  as 
of  fiscal  year  2005,  the  threshold  is  $10  million  or  20  percent  of  the  program's 
appropriation,  whichever  is  less.  To  fund  shortfalls  in  Navy  ERP,  DON  plans  to  reprogram 
amounts  below  this  threshold  from  other  programs. 

9FOC  means  that  the  system  has  been  successfully  deployed  in  all  intended  locations. 

10This  2003  estimate,  which  was  prepared  to  assist  in  budget  development  and  support  the 
Milestone  A/B  approval,  was  for  development,  deployment,  and  sustainment  costs  in  fiscal 
years  2003  through  2021. 

^According  to  DOD’s  acquisition  guidebook,  an  Acquisition  Program  Baseline  is  a  program 
manager’s  estimated  cost,  schedule,  and  performance  goals.  Goals  consist  of  objective 
values,  which  represent  what  the  user  desires  and  expects,  and  threshold  values,  which 
represent  acceptable  limits.  When  the  program  manager  determines  a  current  cost, 
schedule  or  perfonnance  threshold  value  will  not  be  achieved,  the  MDA  must  be  notified, 
and  a  new  baseline  developed,  reviewed  by  decision  makers,  and,  if  the  program  is  to 
continue,  approved  by  the  MDA. 

12According  to  the  August  2004  Acquisition  Program  Baseline,  this  estimate  is  for 
acquisition,  operations,  and  support  for  fiscal  years  2004  through  2021. 
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estimate).13  Key  activities  for  each  phase  are  depicted  in  figure  1,  changes 
in  the  deployment  schedule  are  depicted  in  figure  2,  and  cost  estimates  are 
depicted  in  figure  3. 


13According  to  the  September  2007  Acquisition  Program  Baseline,  this  estimate  is  for 
acquisition,  operations,  and  support  for  fiscal  years  2004  through  2023. 
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Figure  1:  Navy  ERP  Time  Line 
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Source:  GAO  analysis  of  DON  data. 
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Figure  2:  Navy  ERP  Milestones  for  Beginning  Deployment 
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Figure  3:  Navy  ERP  Life  Cycle  Cost  Estimates  in  Fiscal  Years  2003,  2004,  and  2007 
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Use  of  IT  Acquisition 
Management  Controls 
Maximizes  Chances  for 
Success 


IT  acquisition  management  controls  are  tried  and  proven  methods, 
processes,  techniques,  and  activities  that  organizations  define  and  use  to 
minimize  program  risks  and  maximize  the  chances  of  a  program’s  success. 
Using  these  controls  can  result  in  better  outcomes,  including  cost  savings, 
improved  service  and  product  quality,  and  a  better  return  on  investment. 
For  example,  two  software  engineering  analyses  of  nearly  200  systems 
acquisition  projects  indicate  that  teams  using  systems  acquisition  controls 
that  reflected  best  practices  produced  cost  savings  of  at  least  11  percent 
over  similar  projects  conducted  by  teams  that  did  not  employ  the  kind  of 
rigor  and  discipline  embedded  in  these  practices.14  In  addition,  our 
research  shows  that  these  controls  are  a  significant  factor  in  successful 
acquisition  outcomes,  including  increasing  the  likelihood  that  programs 
and  projects  will  be  executed  within  cost  and  schedule  estimates.15 

We  and  others  have  identified  and  promoted  the  use  of  a  number  of  IT 
acquisition  management  controls  associated  with  acquiring  IT  systems.16 
See  table  4  for  a  description  of  several  of  these  activities. 


14Donald  E.  Harter,  Mayuram  S.  Krishnan,  and  Sandra  A.  Slaughter,  “Effects  of  Process 
Maturity  on  Quality,  Cycle  Time,  and  Effort  in  Software  Product  Development,” 
Management  Science ,  46,  no.  4  (2000);  and  Bradford  K.  Clark,  “Quantifying  the  Effects  of 
Process  Improvement  on  Effort,”  IEEE  Software  (November/December  2000). 

15GAO,  Information  Technology:  DOD’s  Acquisition  Policies  and  Guidance  Need  to 
Incorporate  Additional  Best  Practices  and  Controls,  GAO-04-722  (Washington,  D.C.: 

July  30,  2004). 

16GAO-04-722. 
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Table  4:  Description  of  Business  System  IT  Acquisition  Management  Controls 

Business  system  acquisition  practice 

Description 

Architectural  alignment 

To  ensure  that  the  acquisition  is  consistent 
with  the  organization’s  enterprise 
architecture. 

Architectural  alignment  is  the  process  for  analyzing  and  verifying  that  the  proposed 
architecture  of  the  system  being  acquired  is  consistent  with  the  enterprise  architecture 
for  the  organization  acquiring  the  system.  Such  alignment  is  needed  to  ensure  that 
acquired  systems  can  interoperate  and  are  not  unnecessarily  duplicative  of  one  another. 

Economic  justification 

To  ensure  that  system  investments  are 
economically  justified. 

Economic  justification  is  the  process  for  ensuring  that  acquisition  decisions  are  based 
on  reliable  analyses  of  the  proposed  investment’s  likely  costs  versus  benefits  over  its 
useful  life,  as  well  as  an  analysis  of  the  risks  associated  with  actually  realizing  the 
acquisition’s  forecasted  benefits  for  its  estimated  costs.  Economic  justification  is  not  a 
one-time  event  but  rather  is  performed  throughout  an  acquisition’s  life  cycle  in  order  to 
permit  informed  investment  decision  making. 

Earned  value  management 

To  ensure  that  actual  progress  against  cost 
and  schedule  expectations  is  being 
measured. 

EVM  is  a  tool  that  integrates  the  technical,  cost,  and  schedule  parameters  of  a  contract 
and  measures  progress  against  them.  During  the  planning  phase,  an  integrated 
program  baseline  is  developed  by  time  phasing  budget  resources  for  defined  work.  As 
work  is  performed  and  measured  against  the  baseline,  the  corresponding  budget  value 
is  “earned.”  Using  this  earned  value  metric,  cost  and  schedule  variances,  as  well  as  cost 
and  time  to  complete  estimates,  can  be  determined  and  analyzed. 

Requirements  management 

To  ensure  that  requirements  are  traceable, 
verifiable,  and  controlled. 

Requirements  management  is  the  process  for  ensuring  that  the  requirements  are 
traceable,  verifiable,  and  controlled.  Traceability  refers  to  the  ability  to  follow  a 
requirement  from  origin  to  implementation  and  is  critical  to  understanding  the 
interconnections  and  dependencies  among  the  individual  requirements  and  the  impact 
when  a  requirement  is  changed.  Requirements  management  begins  when  the 
solicitation’s  requirements  are  documented  and  ends  when  system  responsibility  is 
transferred  to  the  support  organization. 

Risk  management 

To  ensure  that  risks  are  identified  and 
systematically  mitigated. 

Risk  management  is  the  process  for  identifying  potential  acquisition  problems  and  taking 
appropriate  steps  to  avoid  their  becoming  actual  problems.  Risk  management  occurs 
early  and  continuously  in  the  acquisition  life  cycle. 

Source:  GAO. 
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Prior  GAO  Reviews  Have 
Identified  IT  Acquisition 
Management  Control 
Weaknesses  on  DOD 
Business  System 
Investments 


We  have  previously  reported17  that  DOD  has  not  effectively  managed  a 
number  of  business  system  investments.  Among  other  things,  our  reviews 
of  individual  system  investments  have  identified  weaknesses  in  such 
things  as  architectural  alignment  and  informed  investment  decision 
making,  which  are  also  the  focus  areas  of  the  Fiscal  Year  2005  Defense 
Authorization  Act  business  system  provisions.18  Our  reviews  have  also 
identified  weaknesses  in  other  system  acquisition  and  investment 
management  areas — such  as  EVM,  economic  justification,  requirements 
management,  and  risk  management. 


In  July  2007,  we  reported  that  the  Army’s  approach  for  investing  about  $5 
billion  over  the  next  several  years  in  its  General  Fund  Enterprise  Business 
System,  Global  Combat  Support  System-Army  Field/Tactical,19  and  Logistics 
Modernization  Program  did  not  include  alignment  with  the  Army  enterprise 
architecture  or  use  of  a  portfolio-based  business  system  investment  review 
process.20  Moreover,  we  reported  that  the  Army  did  not  have  reliable  analyses, 
such  as  economic  analyses,  to  support  its  management  of  these  programs.  We 
concluded  that,  until  the  Army  adopts  a  business  system  investment 
management  approach  that  provides  for  reviewing  groups  of  systems  and 
making  enterprise  decisions  on  how  these  groups  will  collectively  interoperate 
to  provide  a  desired  capability,  it  runs  the  risk  of  investing  significant  resources 
in  business  systems  that  do  not  provide  the  desired  functionality  and  efficiency. 
Accordingly,  we  made  recommendations  aimed  at  improving  the  department’s 
efforts  to  achieve  total  asset  visibility  and  enhancing  its  efforts  to  improve  its 
control  and  accountability  over  business  system  investments.  The  department 
agreed  with  our  recommendations. 


17See,  for  example,  GAO,  DOD  Business  Transformation:  Lack  of  an  Integrated  Strategy 
Puts  the  Army’s  Asset  Visibility  System  Investments  at  Risk,  GAO-07-860  (Washington, 
D.C.:  July  27,  2007);  GAO,  Information  Technology:  DOD  Needs  to  Ensure  That.  Navy 
Marine  Corps  Intranet  Program  Is  Meeting  Goals  and  Satisfying  Customers,  GAO-07-51 
(Washington,  D.C.:  Dec.  8,  2006);  GAO,  Defense  Travel  System:  Reported  Savings 
Questionable  and  Implementation  Challenges  Remain,  GAO-06-980  (Washington,  D.C.: 
Sept.  26,  2006);  GAO,  DOD  Systems  Modernization:  Uncertain  Joint  Use  and  Marginal 
Expected  Value  of  Military  Asset  Deployment  System  Warrant  Reassessment  of  Planned 
Investment,  GAO-06-171  (Washington,  D.C.:  Dec.  15,  2005);  and  GAO,  DOD  Systems 
Modernization:  Planned  Investment  in  the  Navy  Tactical  Command  Support  System 
Needs  to  Be  Reassessed,  GAO-06-215  (Washington,  D.C.:  Dec.  5,  2005). 

lsRonald  W.  Reagan  National  Defense  Authorization  Act  for  Fiscal  Year  2005,  Pub.  L. 

No.  108-375,  §  332  (2004)  (codified  at  10  U.S.C.  §§  186  and  2222). 

19Field/tactical  refers  to  Army  units  that  are  deployable  to  locations  around  the  world,  such 
as  Iraq  or  Afghanistan. 

zoGAO-07-860. 
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We  also  reported  that  DON  had  not,  among  other  things,  economically 
justified  its  ongoing  and  planned  investment  in  the  Naval  Tactical 
Command  Support  System  (NTCSS)21  and  had  not  invested  in  NTCSS 
within  the  context  of  a  well-defined  DOD  or  DON  enterprise  architecture. 
In  addition,  we  reported  that  DON  had  not  effectively  performed  key 
measurement,  reporting,  budgeting,  and  oversight  activities  and  had  not 
adequately  conducted  requirements  management  and  testing  activities.  We 
concluded  that,  without  this  information,  DON  could  not  determine 
whether  NTCSS,  as  defined,  and  as  being  developed,  is  the  right  solution 
to  meet  its  strategic  business  and  technological  needs.  Accordingly,  we 
recommended  that  the  department  develop  the  analytical  basis  to 
determine  if  continued  investment  in  NTCSS  represents  prudent  use  of 
limited  resources  and  to  strengthen  management  of  the  program, 
conditional  upon  a  decision  to  proceed  with  further  investment  in  the 
program.  The  department  largely  agreed  with  our  recommendations. 

In  addition,  we  reported  that  the  Army  had  not  defined  and  developed  its 
Transportation  Coordinators’  Automated  Information  for  Movements  System 
II  (TC-AJMS II) — a  joint  services  system  with  the  goal  of  helping  to  manage 
the  movement  of  forces  and  equipment  within  the  United  States  and  abroad — 
in  the  context  of  a  DOD  enterprise  architecture.22  In  addition,  we  reported 
that  the  Army  had  not  economically  justified  the  program  on  the  basis  of 
reliable  estimates  of  life  cycle  costs  and  benefits  and  had  not  effectively 
implemented  risk  management.  As  a  result,  we  concluded  that  the  Army  did 
not  know  if  its  investment  in  TC-AIMS II,  as  planned,  was  warranted  or 
represented  a  prudent  use  of  limited  DOD  resources.  Accordingly,  we 
recommended  that  the  department,  among  other  things,  develop  the 
analytical  basis  needed  to  determine  if  continued  investment  in  TC-AIMS  II 
represents  prudent  use  of  limited  defense  resources.  In  response,  the 
department  agreed  with  our  recommendations  and  has  since  reduced  the 
program’s  scope  by  canceling  future  investments. 

Furthermore,  in  2005,  we  reported  that  DON  had  invested  approximately 
$1  billion  in  the  four  previously  cited  ERP  pilots  without  marked 
improvement  in  its  day-to-day  operations.23  More  specifically,  we  reported 


21GAO-06-215. 

22GAO-06-171. 

23GAO,  DOD  Business  Systems  Modernization:  Navy  ERP  Adherence  to  Best  Business 
Practices  Critical  to  Avoid  Past  Failures,  GAO-05-858  (Washington,  D.C.:  Sept.  29,  2005). 
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that  the  program  office  had  not  implemented  an  EVM  system.  We  also 
identified  significant  challenges  and  risks  as  the  project  moved  forward, 
such  as  developing  and  implementing  system  interfaces,  converting  data 
from  legacy  systems  into  the  ERP  system,  meeting  its  estimated 
completion  date  of  2011  at  an  estimated  cost  of  $800  million,  and  achieving 
alignment  with  DOD’s  BEA.  To  address  these  areas,  we  made 
recommendations  that  DOD  improve  oversight  of  Navy  ERP,  including 
developing  quantitative  metrics  to  evaluate  the  program.  DOD  generally 
agreed  with  our  recommendations. 


Implementation  of 
Key  DOD  and  Related 
IT  Acquisition 
Management  Controls 
on  Navy  ERP  Varies 


DOD  IT-related  acquisition  policies  and  guidance,  along  with  other 
relevant  guidance,  provide  an  acquisition  management  control  framework 
within  which  to  manage  business  system  programs  like  Navy  ERP. 
Effective  implementation  of  this  framework  can  minimize  program  risks 
and  better  ensure  that  system  investments  are  defined  in  a  way  to 
optimally  support  mission  operations  and  performance,  as  well  as  deliver 
promised  system  capabilities  and  benefits  on  time  and  within  budget.  To 
varying  degrees  of  effectiveness,  Navy  ERP  has  been  managed  in 
accordance  with  aspects  of  this  framework.  However,  implementation  of 
key  management  controls  has  not  been  effective.  Specifically, 


•  compliance  with  DOD’s  federated  BEA  has  not  been  sufficiently 
demonstrated; 

•  investment  in  the  program  has  been  economically  justified  on  the  basis  of 
expected  life  cycle  benefits  that  will  likely  exceed  estimated  life  cycle 
costs,  although  some  estimating  limitations  nevertheless  exist; 

•  earned  value  management  has  not  been  effectively  implemented; 

•  an  important  requirements  management  activity  has  been  effectively 
implemented;  and 

•  a  risk  management  process  has  been  defined,  but  not  effectively 
implemented  for  all  risks. 


The  reasons  that  program  management  and  oversight  officials  cited  for 
why  these  key  practices  have  not  been  sufficiently  executed  range  from 
limitations  in  the  applicable  DOD  guidance  and  tools  to  the  complexity 
and  challenges  of  managing  and  implementing  a  program  of  this  size.  Each 
of  these  reasons  is  described  in  the  applicable  sections  of  this  report.  By 
not  effectively  implementing  all  the  above  key  IT  acquisition  management 
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functions,  the  program  is  at  increased  risk  of  (1)  not  being  defined  in  a 
way  that  best  meets  corporate  mission  needs  and  enhances  performance 
and  (2)  adding  to  the  more  than  2  years  in  program  schedule  delays  and 
about  $570  million  in  program  cost  increases  experienced  to  date. 


Navy  ERP  Compliance 
with  DOD’s  Federated  BEA 
Has  Not  Been  Sufficiently 
Demonstrated 


DOD  and  other  guidance,24  recognize  the  importance  of  investing  in 
business  systems  within  the  context  of  an  enterprise  architecture.26 
Moreover,  the  Fiscal  Year  2005  Defense  Authorization  Act  requires  that 
defense  business  systems  be  compliant  with  DOD’s  federated  BEA.26  Our 
research  and  experience  in  reviewing  federal  agencies  show  that  not 
making  investments  within  the  context  of  a  well-defined  enterprise 
architecture  often  results  in  systems  that  are  duplicative,  are  not  well 


24Department  of  Defense  Architecture  Framework,  Version  1.0,  Volume  1  (February  2004); 
GAO,  Information  Technology:  A  Framework  for  Assessing  and  Improving  Enterprise 
Architecture  Management  (Version  1.1),  GAO-03-584G  (Washington,  D.C.:  Apr.  1,  2003); 
Chief  Information  Officer  Council,  A  Practical  Guide  to  Federal  Enterprise  Architecture, 
Version  1.0  (February  2001);  and  Institute  of  Electrical  and  Electronics  Engineers  (IEEE), 
Standard  for  Recommended  Practice  for  Architectural  Description  of  Software-Intensive 
Systems  1471-2000  (Sept.  21,  2000). 

25A  well-defined  enterprise  architecture  provides  a  clear  and  comprehensive  picture  of  an 
entity,  whether  it  is  an  organization  (e.g.,  a  federal  department)  or  a  functional  or  mission 
area  that  cuts  across  more  than  one  organization  (e.g.,  personnel  management).  This 
picture  consists  of  snapshots  of  both  the  enterprise’s  current  or  “as  is”  environment  and  its 
target  or  “to  be”  environment,  as  well  as  a  capital  investment  road  map  for  transitioning 
from  the  current  to  the  target  environment.  These  snapshots  consist  of  integrated  “views,” 
which  are  one  or  more  architecture  products  that  describe,  for  example,  the  enterprise’s 
business  processes  and  rules;  information  needs  and  flows  among  functions,  supporting 
systems,  services,  and  applications;  and  data  and  technical  standards  and  structures. 

26DOD  has  adopted  a  federated  approach  for  developing  its  business  mission  area 
enterprise  architecture,  which  includes  the  corporate  BEA  representing  the  thin  layer  of 
DOD-wide  corporate  architectural  policies,  capabilities,  rules,  and  standards;  component 
architectures  (e.g.,  Navy  enterprise  architecture);  and  program  architectures  (e.g.,  Navy 
ERP  architecture). 
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integrated,  are  unnecessarily  costly  to  interface  and  maintain,  and  do  not 
optimally  support  mission  outcomes.27 

To  its  credit,  the  program  office  has  followed  DOD’s  BEA  compliance 
guidance.28  However,  this  guidance  does  not  adequately  provide  for 
addressing  all  relevant  aspects  of  BEA  compliance.  Moreover,  DON’s 
enterprise  architecture,  which  is  a  major  component  of  DOD’s  federated 
BEA,  as  well  as  key  aspects  of  DOD’s  corporate  BEA,  have  yet  to  be 
sufficiently  defined  to  permit  thorough  compliance  determinations.  In 
addition,  current  policies  and  guidance  do  not  require  DON  investments  to 
comply  with  its  enterprise  architecture.  This  means  that  the  department 
does  not  have  a  sufficient  basis  for  knowing  if  Navy  ERP  has  been  defined 
to  minimize  overlap  with  and  duplication  of  other  programs’  functionality 
and  maximize  interoperability  among  related  programs.  Each  of  these 
architecture  alignment  limitations  is  discussed  here: 

•  The  program’s  compliance  assessments  did  not  include  all  relevant 
architecture  products.  In  particular,  the  program  did  not  assess 
compliance  with  the  BEA’s  technical  standards  profile,  which  outlines,  for 
example,  the  standards  governing  how  systems  physically  communicate 
with  other  systems  and  how  they  secure  data  from  unauthorized  access. 
This  is  particularly  important  because  systems  like  Navy  ERP  need  to 
share  information  with  other  systems  and,  for  these  systems  to  accomplish 
this  effectively  and  efficiently,  they  need  to  employ  common  standards.  A 
case  in  point  is  the  relationship  between  Navy  ERP  and  the  Global  Combat 


27See,  for  example,  GAO,  Information  Technology:  FBI  Is  Taking  Steps  to  Develop  an 
Enterprise  Architecture,  but  much  Remains  to  Be  Accomplished,  GAO-05-363 
(Washington,  D.C.:  Sept.  9,  2005);  GAO,  Homeland  Security:  Efforts  Under  Way  to  Develop 
Enterprise  Architecture,  but  Much  Work  Remains,  GAO-04-777  (Washington,  D.C.:  Aug.  6, 
2004);  GAO ,  Information  Technology:  Architecture  Needed  to  Guide  NASA’s  Financial 
Management  Modernization,  GAO-04-43  (Washington,  D.C.:  Nov.  21,  2003);  GAO,  DOD 
Business  Systems  Modernization:  Important  Progress  Made  to  Develop  Business 
Enterprise  Architecture,  but  Much  Work  Remains,  GAO-03-1018  (Washington,  D.C.: 

Sept.  19,  2003);  GAO,  Information  Technology:  DLA  Should  Strengthen  Business  Systems 
Modernization  Architecture  and  Investment  Activities,  GAO-Ol-631  (Washington,  D.C.: 
June  29,  2001);  and  GAO,  Information  Technology:  INS  Needs  to  Better  Manage  the 
Development  of  Its  Enterprise  Architecture,  GAO/AIMD-OO-212  (Washington,  D.C.: 

Aug.  1,  2000). 

28DOD,  Business  Enterprise  Architecture  Compliance  Guidance  (Apr.  10,  2006). 


Page  23 


GAO-08-896  DOD  Business  Systems  Modernization 


Support  System — Marine  Corps  (GCSS-MC)  program.29  Specifically,  Navy 
ERP  has  identified  25  technical  standards  that  are  not  in  the  BEA  technical 
standards  profile,  and  GCSS-MC  has  identified  13  technical  standards  that 
are  not  in  the  profile.  Among  these  non-BEA  standards  are  program-unique 
information  sharing  protocols,  which  could  limit  information  sharing 
between  Navy  ERP  and  GCSS-MC,  and  with  other  systems. 

In  addition,  the  program  office  did  not  assess  compliance  with  the  BEA 
products  that  describe  system-level  characteristics.  This  is  important 
because  doing  so  would  create  a  body  of  information  about  programs  that 
could  be  used  to  identify  common  system  components  and  services  that 
could  potentially  be  shared  by  the  programs,  thus  avoiding  wasteful 
duplication.  For  example,  our  analysis  of  Navy  ERP  program 
documentation  shows  that  it  contains  system  functions  related  to 
receiving  goods,  taking  physical  inventories,  and  returning  goods,  which 
are  also  system  functions  cited  by  the  GCSS-MC  program.  However, 
because  compliance  with  the  BEA  system  products  was  not  assessed,  the 
extent  to  which  these  functions  are  potentially  duplicative  was  not 
considered. 

Furthermore,  the  program  office  did  not  assess  compliance  with  BEA 
system  products  that  describe  data  exchanges  among  systems.  As  we 
previously  reported,  establishing  and  using  standard  system  interfaces  is  a 
critical  enabler  to  sharing  data.30  For  example,  Navy  ERP  program 
documentation  indicates  that  it  is  to  exchange  inventory  order  and  status 
data  with  other  systems.  System  interfaces  are  important  for 
understanding  how  information  is  to  be  exchanged  between  systems. 
However,  since  the  program  was  not  assessed  for  compliance  with  these 
products,  it  does  not  have  the  basis  for  understanding  how  its  approach  to 
exchanging  information  differs  from  that  of  other  systems  that  it  is  to 
interface  with.  Compliance  against  each  of  these  BEA  products  was  not 
assessed  because  DOD’s  compliance  guidance  does  not  provide  for  doing 


"initiated  in  2003,  GCSS-MC  is  to  modernize  the  Marine  Corps  logistics  systems  and 
thereby  provide  the  decision  makers  with  timely  and  complete  logistics  information  to 
support  the  warfighter.  Moreover,  according  to  program  officials,  both  GCSS-MC  and  Navy 
ERP  are  under  the  leadership  of  Navy’s  Program  Executive  Office  for  Enterprise 
Information  Systems,  which  is  responsible  for  developing,  acquiring,  and  deploying 
seamless  enterprise-wide  information  technology  systems. 

30GAO,  DOD  Business  Systems  Modernization:  Progress  in  Establishing  Corporate 
Management  Controls  Needs  to  be  Replicated  Within  Military  Departments,  GAO-08-705 
(Washington,  D.C.:  May  15,  2008). 
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so  and,  according  to  BTA  officials,  because  some  BEA  system  products 
are  not  sufficiently  defined.  According  to  these  officials,  BTA  plans  to 
continue  to  define  these  products  as  the  BEA  evolves. 

•  The  compliance  assessment  was  not  used  to  identify  potential  areas  of 
duplication  across  programs,  which  DOD  has  stated  is  an  explicit  goal  of 
its  federated  BEA  and  associated  investment  review  and  decision-making 
processes.  More  specifically,  even  though  the  compliance  guidance 
provides  for  assessing  programs’  compliance  with  the  BEA  product  that 
defines  DOD  operational  activities,  and  Navy  ERP  was  assessed  for 
compliance  with  this  product,  the  results  were  not  used  to  identify 
programs  that  support  the  same  operational  activities  and  related  business 
processes.  Given  that  the  federated  BEA  is  intended  to  identify  and  avoid 
not  only  duplications  within  DOD  components,  but  also  between 
components,  it  is  important  that  such  commonality  be  addressed.  For 
example,  BEA  compliance  assessments  for  Navy  ERP  and  GCSS-MC,  as 
well  as  two  Air  Force  programs  (Defense  Enterprise  Accounting  and 
Management  System — Air  Force  and  the  Air  Force  Expeditionary  Combat 
Support  System)  show  that  each  program  supports  at  least  six  of  the  same 
BEA  operational  activities  (e.g.,  conducting  physical  inventory,  delivering 
property  and  services)  and  three  of  these  four  programs  support  at  least 
18  additional  operational  activities  (e.g.,  performing  budgeting,  managing 
receipt  and  acceptance).  However,  since  the  potential  overlap  among 
these  and  other  programs  was  not  assessed,  these  programs  may  be 
investing  in  duplicative  functionality.  Reasons  for  this  were  that  the 
compliance  guidance  does  not  provide  for  such  analyses  to  be  conducted 
and  programs  have  not  been  granted  access  rights  to  use  this  functionality 
in  the  compliance  tool. 

•  The  program’s  compliance  assessment  did  not  address  compliance  against 
DON’s  enterprise  architecture,  which  is  one  of  the  biggest  members  of  the 
federated  BEA.  This  is  particularly  important  given  that  DOD’s  approach 
to  fully  satisfying  the  architecture  requirements  of  the  Fiscal  Year  2005 
Defense  Authorization  Act  is  to  develop  and  use  a  federated  architecture 
in  which  component  architectures  are  to  provide  the  additional  details 
needed  to  supplement  the  thin  layer  of  corporate  policies,  rales,  and 
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standards  included  in  the  corporate  BEA.31  As  we  recently  reported,32 
DON’s  enterprise  architecture  is  not  mature  because,  among  other  things, 
it  is  missing  a  sufficient  description  of  its  current  and  future  environments 
in  terms  of  business  and  information/data.  However,  certain  aspects  of  an 
architecture  nevertheless  exist  and,  according  to  DON  CIO  officials,  these 
aspects  will  be  leveraged  in  its  efforts  to  develop  a  complete  enterprise 
architecture.  For  example,  the  FORCEnet  architecture  is  intended  to 
document  Navy’s  technical  infrastructure.  Therefore,  opportunities  exist 
for  DON  to  assess  its  programs  in  relation  to  these  architecture  products, 
and  to  understand  where  its  programs  are  exposed  to  risks  because 
products  do  not  exist,  are  not  mature,  or  at  odds  with  other  Navy 
programs.  According  to  DOD  officials,  compliance  with  the  DON 
architecture  was  not  assessed  because  DOD  compliance  policy  is  limited 
to  compliance  with  the  corporate  BEA,  and  a  number  of  aspects  of  the 
DON  enterprise  architecture  have  yet  to  be  sufficiently  developed. 

•  The  program’s  compliance  assessment  was  not  validated  by  DON  or  DOD 
investment  oversight  and  decision-making  authorities.  More  specifically, 
neither  the  DOD  IRBs  nor  the  DBSMC,  nor  the  BTA  in  supporting  both  of 
these  investment  oversight  and  decision-making  authorities,  reviewed  the 
program’s  assessments.  According  to  BTA  officials,  under  DOD’s  tiered 
approach  to  investment  accountability,  these  entities  are  not  responsible 
for  validating  programs’  compliance  assessments.  Rather,  this  is  a 
component  responsibility,  and  thus  they  rely  on  the  military  departments 
and  defense  agencies  to  validate  the  assessments. 

However,  DON  Office  of  the  CIO,  which  is  responsible  for  precertifying 
investments  as  compliant  before  they  are  reviewed  by  the  IRB,  did  not 
validate  any  of  the  program’s  compliance  assessments.  According  to  Office 
of  the  CIO  officials,  they  rely  on  Functional  Area  Managers  to  validate  a 
program’s  compliance  assessments.  However,  no  DON  policy  or  guidance 
exists  that  describes  how  the  Functional  Area  Managers  should  conduct 
such  validations.  CIO  officials  stated  that  this  is  because  these  authorities 
do  not  have  the  resources  that  they  need  to  validate  the  assessments,  and 


31As  we  recently  reported,  while  the  corporate  BEA  includes  corporate  policies, 
capabilities,  rules,  and  standards,  it  is  still  evolving  and  will  continue  to  add  additional 
details.  See  GAO-08-705. 

32GAO,  DOD  Business  Systems  Modernization:  Military  Departments  Need  to  Strengthen 
Management  of  Enterprise  Architecture  Programs,  GAO-08-519  (Washington,  D.C.: 

May  12,  2008). 
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because  a  number  of  aspects  of  the  DON  architecture  are  not  yet 
sufficiently  developed. 

Validation  of  program  assessments  is  further  complicated  by  the  absence 
of  information  captured  in  the  assessment  tool  about  what  program 
documentation  or  other  source  materials  were  used  by  the  program  office 
in  making  its  compliance  determinations.  Specifically,  the  tool  is  only 
configured,  and  thus  was  only  used,  to  capture  the  results  of  a  program’s 
comparison  of  program  architecture  products  to  BEA  products.  Thus,  it 
was  not  used  to  capture  the  system  products  used  in  making  these 
determinations. 

The  limitations  in  existing  BEA  compliance-related  policy  and  guidance, 
the  supporting  compliance  assessment  tool,  and  the  federated  BEA,  put 
programs  like  Navy  ERP  at  increased  risk  of  being  defined  and 
implemented  in  a  way  that  does  not  sufficiently  ensure  interoperability 
and  avoid  duplication  and  overlap.  We  recently  completed  a  review 
examining  multiple  programs’  compliance  with  the  federated  BEA, 
including  Navy  ERP,  for  the  Senate  Armed  Services  Committee, 
Subcommittee  on  Readiness  and  Management  Support.  We  addressed  the 
architectural  compliance  guidance,  tool,  and  validation  limitations  as  part 
of  this  review.33 


Investment  in  Navy  ERP 
Has  Been  Economically 
Justified,  but  Important 
Estimating  Practices  Were 
Not  Implemented 


The  investment  in  Navy  ERP  has  been  economically  justified  on  the  basis 
of  expected  life  cycle  benefits  that  far  exceed  estimated  life  cycle  costs. 
According  to  the  program’s  benefit/cost  analysis,  Navy  ERP  will  produce 
about  $8.6  billion  in  estimated  benefits  for  an  estimated  cost  of  about  $2.4 
billion  over  its  20-year  life  cycle.  While  these  benefit  estimates  were  not 
subject  to  any  analysis  of  how  uncertainty  in  assumptions  and  data  could 
impact  the  estimates,  as  called  for  by  relevant  guidance,  our  examination 
of  key  uncertainty  variables,  such  as  the  timing  of  legacy  systems’ 
retirement,  showed  that  the  savings  impact  would  be  relatively  minor. 
However,  the  reliability  of  the  cost  estimate  is  limited  because  it  was 
derived  using  several,  but  not  all,  key  estimating  practices.  For  example, 
the  estimate  was  not  grounded  in  a  historical  record  of  comparable  data 
from  similar  programs  and  was  not  based  on  a  reliable  schedule  baseline, 


33GAO,  DOD  Business  Systems  Modernization:  Key  Navy  Programs’  Compliance  ivith 
DOD’s  Federated  Business  Enterprise  Architecture  Needs  to  Be  Adequately 
Demonstrated,  GAO-08-972  (Washington,  D.C.:  Aug.  7,  2008). 
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Benefit  Estimates  Are 
Sufficiently  Reliable  Despite 
Absence  of  Uncertainty 
Analysis 


which  are  both  necessary  to  having  a  cost  estimate  that  can  be  considered 
credible  and  accurate.  These  practices  were  not  employed  for  various 
reasons,  including  DOD’s  lack  of  historical  data  from  similar  programs  and 
the  lack  of  an  integrated  master  schedule  for  the  program  that  includes  all 
releases. 

Notwithstanding  the  fact  that  these  limitations  could  materially  increase 
the  $2.4  billion  cost  estimate,  it  is  nevertheless  unlikely  that  these  factors 
would  increase  the  estimate  to  a  level  approaching  the  program’s  benefit 
expectations.  Therefore,  we  have  no  reason  to  believe  that  Navy  ERP  will 
not  produce  a  positive  return  on  investment. 

Forecasting  expected  benefits  over  the  life  of  a  program  is  a  key  aspect  of 
economically  justifying  an  investment.  The  Office  of  Management  and 
Budget  (OMB)  guidance34  advocates  economically  justifying  investments 
on  the  basis  of  expected  benefits,  costs,  and  risks.  Since  estimates  of 
benefits  can  be  uncertain  because  of  the  imprecision  in  both  the 
underlying  data  and  modeling  assumptions  used,  the  guidance  also 
provides  for  analyzing  and  reporting  the  effects  of  this  uncertainty.  By 
doing  this,  informed  investment  decision  making  can  occur  through  the 
life  of  the  program,  and  a  baseline  can  be  established  against  which  to 
compare  the  accrual  of  actual  benefits  from  deployed  system  capabilities. 

The  most  recent  economic  analysis,  dated  August  2007,  includes 
monetized  benefit  estimates  for  fiscal  years  2004-2023,  in  three  key 
areas — about  $2.7  billion  in  legacy  system  cost  savings,  $3.3  billion  in  cost 
savings  from  inventory  reductions,  and  $2.7  billion  in  cost  savings  from 
labor  productivity  improvements.  Collectively,  these  benefits  total  about 
$8.6  billion.35 

The  program  office  calculated  expected  benefits  in  terms  of  cost  savings, 
which  is  consistent  with  established  practices  and  guidance.  For  example, 
the  program  is  to  result  in  the  retirement  of  138  legacy  systems  (including 
the  4  pilot  systems)  between  fiscal  years  2006  and  2015,  and  the  yearly 
maintenance  costs  for  a  single  system  are  expected  to  be  as  high  as  about 


340ffice  of  Management  and  Budget,  Circular  No.  A-94:  Guidelines  and  Discount  Rates  for 
Benefit-Cost  Analysis  of  Federal  Programs  (Oct.  29,  1992). 

35The  benefits  estimates  for  the  areas  are  rounded;  therefore,  they  add  to  more  than 
$8.6  billion. 
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The  Cost  Estimate  Was  Not 
Reliably  Derived 


$39  million.36  According  to  relevant  guidance,  cost  saving  estimates  should 
also  be  analyzed  in  terms  of  how  uncertainty  in  assumptions  and  data 
could  impact  them.  However,  the  program  office  did  not  perform  such 
uncertainty  analysis.  According  to  program  officials,  uncertainty  analysis 
is  not  warranted  because  they  have  taken  and  continue  to  take  steps  to 
validate  the  assumptions  and  the  data,  such  as  using  the  latest  budget  data 
associated  with  the  legacy  systems,  and  monitoring  changes  to  the 
systems’  retirement  dates.  While  these  steps  are  positive,  they  do  not 
eliminate  the  need  for  uncertainty  analysis.  Accordingly,  we  assessed  key 
uncertainty  variables,  such  as  the  timing  of  the  legacy  systems’  retirement, 
and  found  that  the  retirement  dates  of  some  of  these  systems  have 
changed  since  the  estimate  was  prepared,  due  to,  among  other  things, 
schedule  delays  in  the  program.  While  the  inherent  uncertainty  in  these 
dates  would  reduce  expected  savings  (e.g.,  only  $11  million  based  on  the 
134  legacy  systems  that  we  examined),37  the  reduction  would  be  small 
relative  to  a  total  benefit  estimate  of  $8.6  billion. 

A  reliable  cost  estimate  is  a  key  variable  in  calculating  return  on 
investment,  and  it  provides  the  basis  for  informed  investment  decision 
making,  realistic  budget  formulation  and  program  resourcing,  meaningful 
progress  measurement,  proactive  course  correction,  and  accountability  for 
results.  According  to  OMB,38  programs  must  maintain  current  and  well- 
documented  cost  estimates,  and  these  estimates  must  encompass  the  full 
life  cycle  of  the  program.  OMB  states  that  generating  reliable  cost 
estimates  is  a  critical  function  necessary  to  support  OMB’s  capital 
programming  process.  Without  reliable  estimates,  programs  are  at 
increased  risk  of  experiencing  cost  overruns,  missed  deadlines,  and 
performance  shortfalls. 

Our  research  has  identified  a  number  of  practices  that  are  the  basis  of 
effective  program  cost  estimating.  We  have  issued  guidance  that 


36For  example,  the  Uniform  Accounting  Data  Process  System-Inventory  Control  Points  has 
yearly  maintenance  costs  of  $39.3  million. 

3 ‘These  reductions  in  expected  savings  represent  the  costs  of  maintaining  the  legacy 
systems  during  the  period  in  which  the  systems’  retirement  dates  have  been  delayed. 

38OMB,  Circular  No.  A-ll,  Preparation,  Submission,  and  Execution  of  the  Budget 
(Washington,  D.C.:  Executive  Office  of  the  President,  June  2006);  Circular  No.  A-130 
Revised,  Management  of  Federal  Information  Resources  (Washington,  D.C.:  Executive 
Office  of  the  President,  Nov.  28,  2000);  and  Capital  Programming  Guide:  Supplement  to 
Circular  A- 1 1 ,  Part  7,  Prepara  tion,  Submission,  and  Execution  of  the  Budget 
(Washington,  D.C.:  Executive  Office  of  the  President,  June  2006). 
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associates  these  practices  with  four  characteristics  of  a  reliable  cost 
estimate.39  Specifically,  these  four  characteristics  are  as  follows: 

•  Comprehensive:  The  cost  estimates  should  include  both  government  and 
contractor  costs  over  the  program’s  full  life  cycle,  from  the  inception  of 
the  program  through  design,  development,  deployment,  and  operation  and 
maintenance  to  retirement.  They  should  also  provide  an  appropriate  level 
of  detail  to  ensure  that  cost  elements  are  neither  omitted  nor  double 
counted  and  include  documentation  of  all  cost-influencing  ground  rules 
and  assumptions. 

•  Well-documented:  The  cost  estimates  should  have  clearly  defined 
purposes  and  be  supported  by  documented  descriptions  of  key  program  or 
system  characteristics  (e.g.,  relationships  with  other  systems,  performance 
parameters).  Additionally,  they  should  capture  in  writing  such  things  as 
the  source  data  used  and  their  significance,  the  calculations  performed 
and  their  results,  and  the  rationale  for  choosing  a  particular  estimating 
method  or  reference.  Moreover,  this  information  should  be  captured  in 
such  a  way  that  the  data  used  to  derive  the  estimate  can  be  traced  back  to, 
and  verified  against,  their  sources.  The  final  cost  estimate  should  be 
reviewed  and  accepted  by  management  on  the  basis  of  confidence  in  the 
estimating  process  and  the  estimate  produced  by  the  process. 

•  Accurate:  The  cost  estimates  should  provide  for  results  that  are  unbiased 
and  should  not  be  overly  conservative  or  optimistic  (i.e.,  they  should 
represent  most  likely  costs).  In  addition,  the  estimates  should  be  updated 
regularly  to  reflect  material  changes  in  the  program,  and  steps  should  be 
taken  to  minimize  mathematical  mistakes  and  their  significance.  Among 
other  things,  the  estimate  should  be  grounded  in  a  historical  record  of  cost 
estimating  and  actual  experiences  on  comparable  programs. 

•  Credible:  The  cost  estimates  should  discuss  any  limitations  in  the  analysis 
performed  due  to  uncertainty  or  biases  surrounding  data  or  assumptions. 
Further,  the  estimates’  derivation  should  provide  for  varying  any  major 
assumptions  and  recalculating  outcomes  based  on  sensitivity  analyses, 
and  their  associated  risks  and  inherent  uncertainty  should  be  disclosed. 
Also,  the  estimates  should  be  verified  based  on  cross-checks  using  other 
estimating  methods  and  by  comparing  the  results  with  independent  cost 
estimates. 


39GAO,  Cost  Assessment  Guide:  Best  Practices  for  Estim  ating  and  Managi  ng  Program 
Costs,  Exposure  Draft,  GAO-07-1134SP  (Washington,  D.C.:  July  2,  2007). 
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The  $2.4  billion  life  cycle  cost  estimate  for  Navy  ERP  reflects  many  of  the 
practices  associated  with  a  reliable  cost  estimate,  including  all  practices 
associated  with  being  comprehensive  and  well-documented,  and  several 
related  to  being  accurate  and  credible  (see  table  5).  However,  several 
important  practices  related  to  accuracy  and  credibility  were  not 
performed.  To  be  reliable,  a  cost  estimate  should  be  comprehensive,  well- 
documented,  accurate,  and  credible. 


Table  5:  Summary  of  Cost  Estimating  Characteristics  That  the  Cost  Estimate 
Satisfies 

Characteristic  of  reliable  estimates 

Satisfied?" 

Comprehensive 

Yes 

Well-documented 

Yes 

Accurate 

Partially 

Credible 

Partially 

Source:  GAO  analysis  of  DON  data. 

"“Yes”  means  that  the  program  office  provided  documentation  demonstrating  satisfaction  of  the 
criterion.  “Partially”  means  that  the  program  office  provided  documentation  demonstrating  satisfaction 
of  part  of  the  criterion.  “No”  means  that  the  program  office  has  yet  to  provide  documentation 
demonstrating  satisfaction  of  the  criterion. 

The  cost  estimate  is  comprehensive  because  it  includes  both  the 
government  and  contractor  costs  specific  to  development,  acquisition 
(nondevelopment),  implementation,  and  operations  and  support  over  the 
program’s  20-year  life  cycle.  Moreover,  the  estimate  clearly  describes  how 
the  various  subelements  are  aggregated  to  produce  amounts  for  each  cost 
category,  thereby  ensuring  that  all  pertinent  costs  are  included,  and  no 
costs  are  double  counted.  Finally,  cost-influencing  ground  rules  and 
assumptions,  such  as  the  program’s  schedule,  labor  rates,  and  inflation 
rates  are  documented. 

The  cost  estimate  is  also  well-documented  in  that  the  purpose  of  the  cost 
estimate  is  clearly  defined,  and  the  technical  baseline  includes,  among 
other  things,  the  hardware  components  and  planned  performance 
parameters.  Furthermore,  the  calculations  and  results  used  to  derive  the 
estimate  are  documented,  including  descriptions  of  the  methodologies 
used  and  evidence  of  traceability  back  to  source  data  (e.g.,  vendor  quotes, 
salary  tables).  Also,  the  cost  estimate  was  reviewed  by  the  Naval  Center 
for  Cost  Analysis  and  the  Office  of  the  Secretary  of  Defense,  Director  for 
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Program  Analysis  and  Evaluation,  which  adds  a  level  of  confidence  in  the 
estimating  process  and  the  estimate  produced. 

However,  the  estimate  lacks  accuracy  because  not  all  important  practices 
related  to  this  characteristic  were  performed.  Specifically,  while  the 
estimate  is  grounded  in  documented  assumptions  (e.g.,  hardware 
refreshment  every  5  years)  and  periodically  updated  to  reflect  changes  to 
the  program,  it  is  not  adequately  grounded  in  historical  experience  with 
comparable  programs.  While  the  program  office  did  leverage  historical 
cost  data  from  the  Navy  ERP  pilot  programs,  program  officials  told  us  that 
the  level  of  cost  accounting  on  these  programs  did  not  provide  sufficient 
data.  As  stated  in  our  guide,  estimates  should  be  based  on  historical 
records  of  cost  and  schedule  estimates  from  comparable  programs,  and 
such  historical  data  should  be  maintained  and  used  for  evaluation 
purposes  and  future  estimates  on  comparable  programs.  The  importance 
of  doing  so  is  evident  by  the  fact  that  Navy  ERP’s  cost  estimate  has 
increased  by  about  $570  million  since  fiscal  year  2003,  which  program 
officials  attributed  to,  among  other  things,  site  implementation  costs  (e.g., 
training  and  converting  legacy  system  data)  not  included  in  the  original 
cost  estimate,  schedule  delays,  and  the  lack  of  historical  data  from  similar 
ERP  programs. 

This  lack  of  cost  data  for  large-scale  ERP  programs  is,  in  part,  due  to  DOD 
not  having  a  standardized  cost  element  structure  for  these  programs  that 
can  be  used  for  capturing  actual  cost  data,  which  is  a  prerequisite  to 
capturing  and  maintaining  the  kind  of  historical  data  that  can  inform  cost 
estimates  on  similar  programs.  This  means  that  programs  like  Navy  ERP 
will  not  be  able  to  ground  their  cost  estimates  in  actual  costs  from 
comparable  programs.  According  to  officials  with  the  Defense  Cost  and 
Resource  Center,40  such  cost  element  structures  are  needed,  along  with  a 
requirement  for  programs  to  report  on  their  costs,  but  approval  and 
resources  have  yet  to  be  gained  for  either  these  structures  or  the  reporting 
of  their  costs.  We  recently  completed  work  that  addressed  standardization 


40The  Defense  Cost  and  Resource  Center  is  responsible  for  collecting  current  and  historical 
major  defense  acquisition  program  cost  and  software  resource  data  in  a  joint  service 
environment  and  making  those  data  available  for  use  by  authorized  government  analysts 
when  estimating  the  cost  of  ongoing  and  future  government  programs. 
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of  DOD’s  ERP  cost  element  structure  and  maintenance  of  a  database  for 
historical  ERP  cost  data  for  use  on  ERP  programs.41 

Compounding  the  estimate’s  limited  accuracy  are  limitations  in  its 
credibility.  Specifically,  while  the  estimate  satisfies  some  of  the  key 
practices  for  a  credible  cost  estimate  (e.g.,  confirming  key  cost  drivers, 
performing  sensitivity  analyses,42  and  having  an  independent  cost  estimate 
prepared  by  the  Naval  Center  for  Cost  Analysis  that  was  within  11  percent 
of  the  program’s  estimate),  the  program  lacks  a  reliable  schedule  baseline, 
which  is  a  key  component  of  a  reliable  cost  estimate  because  it  serves  as 
the  basis  for  future  work  to  be  performed.  Other  factors  that  limit 
confidence  in  the  cost  estimate’s  accuracy  are  (1)  past  increases  in  the 
program’s  cost  estimate  (as  discussed  earlier)  and  (2)  trends  in  EVM  data 
(as  discussed  later).  Taken  together,  the  program’s  cost  estimate  is  not 
sufficiently  credible  and  accurate  and  thus  not  reliable. 

While  important  cost  estimating  practices  were  not  implemented,  it  is 
nevertheless  unlikely  that  these  limitations  would  materially  increase  the 
$2.4  billion  cost  estimate  to  a  level  approaching  the  program’s  $8.6  billion 
benefit  expectations. 


Earned  Value  Management 
Has  Not  Been  Effectively 
Implemented 


Measuring  and  reporting  progress  against  cost  and  schedule  commitments 
(i.e.,  baselines)  is  a  vital  element  of  effective  program  management.  EVM 
provides  a  proven  means  for  measuring  such  progress  and  thereby 
identifying  potential  cost  overruns  and  schedule  delays  early,  when  their 
impact  can  be  minimized. 


To  its  credit,  the  program  has  elected  to  implement  program-level  EVM, 
which  is  a  best  practice  that  has  rarely  been  implemented  in  the  federal 
government.  In  doing  so,  however,  basic  EVM  activities  have  not  been 
executed.  In  particular,  an  integrated  baseline  review,  which  is  to  verify 
that  the  program’s  cost  and  schedule  are  reasonable  given  the  program’s 
scope  of  work  and  associated  risks,  has  not  been  performed.  Moreover, 
other  accepted  industry  standards  have  not  been  sufficiently  implemented, 


41GAO,  DOD  Business  Systems  Modernization:  Key  Marine  Corps  System  Acquisition 
Needs  to  Be  Better  Justified,  Defined,  and  Managed,  GAO-08-822  (Washington,  D.C.: 
July  28,  2008). 

42Sensitivity  analysis  reveals  how  the  cost  estimate  is  affected  by  a  change  in  a  single 
assumption  or  cost  driver  while  holding  all  other  variables  constant. 
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The  Program  Has  Elected  to 
Implement  Program-Level  EVM 


An  Integrated  Baseline  Review 
Has  Not  Been  Performed 


and  surveillance  of  EVM  implementation  by  an  entity  independent  of  the 
program  office  has  not  occurred.  Not  performing  these  important 
practices  has  contributed  to  the  cost  overruns  and  lengthy  schedule  delays 
already  experienced  on  Release  1.0,  and  they  will  likely  result  in  more.  In 
fact,  our  analysis  of  the  latest  estimate  to  complete  just  the  budgeted 
development  work  for  all  three  releases,  which  is  about  $844  million, 
shows  that  this  estimate  will  most  likely  be  exceeded  by  about  $152 
million. 

As  we  previously  reported,43  EVM  offers  many  benefits  when  done 
properly.  In  particular,  it  allows  performance  to  be  measured,  and  it  serves 
as  an  early  warning  system  for  deviations  from  plans.  It,  therefore,  enables 
a  program  office  to  mitigate  the  risks  of  cost  and  schedule  overruns.  OMB 
policy  recognizes  the  use  of  EVM  as  an  important  part  of  program 
management  and  decision  making.44 

Implementing  EVM  at  the  program  level  rather  than  just  the  contract  level 
is  considered  a  best  practice,  and  OMB  recently  began  requiring  it  to 
measure  how  well  a  program’s  approved  cost,  schedule,  and  performance 
goals  are  being  met.  According  to  OMB,  integrating  government  and 
contractor  cost,  schedule,  and  performance  status  should  result  in  better 
program  execution  through  more  effective  management.  In  addition, 
integrated  EVM  data  can  be  used  to  better  justify  budget  requests. 

To  minimize  the  risk  associated  with  its  decision  to  transition 
responsibility  for  Navy  ERP  system  integration  from  the  contractor  to  the 
government  and  to  improve  cost  and  schedule  performance,  the  program 
office  elected  in  October  2006  to  perform  EVM  at  the  program  level.  We 
support  the  use  of  program-level  EVM.  However,  if  not  implemented 
effectively,  this  program-level  approach  will  be  of  little  value. 

A  fundamental  aspect  of  effective  EVM  is  the  development  of  a 
performance  measurement  baseline  (PMB),  which  represents  the 
cumulative  value  of  planned  work  and  serves  as  the  baseline  against  which 


43GAO,  Missile  Defense:  Additional  Knowledge  Needed  in  Developing  System,  for 
Intercepting  Long-Range  Missiles,  GAO-03-600  (Washington,  D.C.:  Aug.  21,  2003). 

44OMB,  Preparation,  Submission,  and  Execution  of  the  Budget,  Circular  A- 11 
(Washington,  D.C.:  Executive  Office  of  the  President,  June  2006),  part  7,  Planning, 
Budgeting,  Acquisition,  and  Management  of  Capital  Assets,  sec.  300. 
http  ://www.  whitehouse .  gov/  omb/circulars/index.html. 
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variances  are  calculated.  According  to  relevant  best  practice  guidance,  a 
PMB  consists  of 

•  a  complete  work  breakdown  structure, 

•  a  complete  integrated  master  schedule,  and 

•  accurate  budgets  for  all  planned  work.45 

To  validate  the  PMB,  an  integrated  baseline  review  is  performed  to  obtain 
stakeholder  agreement  on  the  baseline.  According  to  DOD  guidance  and 
best  practices,  such  a  review  should  be  held  within  6  months  of  a  contract 
award  and  conducted  on  an  as  needed  basis  throughout  the  life  of  a 
program  to  ensure  that  the  baseline  reflects  (1)  all  tasks  in  the  statement 
of  work,  (2)  adequate  resources  (staff  and  materials)  to  complete  the 
tasks,  and  (3)  integration  of  the  tasks  into  a  well-defined  schedule. 

Further,  the  contract  performance  reports  that  are  to  be  used  to  monitor 
performance  against  the  PMB  should  be  validated  during  the  integrated 
baseline  review.46 

The  program  office  has  satisfied  some  of  the  prerequisites  for  having  a 
reliable  PMB,  such  as  developing  a  work  breakdown  structure  and 
specifying  the  contract  performance  reports  that  are  to  be  used  to  monitor 
performance.  However,  it  has  not  conducted  an  integrated  baseline 
review.  Specifically,  a  review  was  not  conducted  for  Release  1.0,  even 
though  the  contract  was  finalized  about  30  months  ago  (January  2006). 
Also,  while  the  review  for  Release  1.1  was  recently  scheduled  for  August 
2008,  this  is  8  months  later  than  when  such  a  review  should  be  held, 
according  to  DOD  guidance  and  best  practices.  This  means  that  the 
reasonableness  of  the  program’s  scope  and  schedule  relative  to  the 
program  risks  has  not  been  assured,  and  has  likely  been,  and  will  likely 
continue  to  be  a  primary  contributor  to  future  cost  increases  and  schedule 
delays. 

According  to  program  officials,  a  review  was  not  performed  on  the  first 
release  because  development  of  this  release  was  largely  complete  by  the 


4BGAO-07-1 134SP. 

46Defense  Contract  Management  Agency,  Department  of  Defense  Earned  Value 
Manaqement  Implementation  Guide  ("Washington,  D.C.:  October  2006);  and 
GAO-07-1 134SP. 
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Industry  EVM  Standards  Have 
Not  Been  Fully  Implemented 
and  Independently  Surveilled 


time  the  program  office  established  the  underlying  capabilities  needed  to 
perform  program-level  EVM.  In  addition,  program  officials  stated  that  an 
integrated  baseline  review  has  yet  to  be  performed  on  the  other  two 
releases  because  their  priority  has  been  on  deploying  and  stabilizing  the 
first  release.  In  our  view,  not  assuring  the  validity  of  the  PMB  precludes 
effective  implementation  of  EVM.  Until  a  review  is  conducted,  DOD  will 
not  have  reasonable  assurance  that  the  program’s  scope  and  schedule  are 
achievable,  and  thus,  additional  cost  and  schedule  overruns  are  likely. 

In  1996,  DOD  adopted  industry  EVM  guidance47  that  identifies  32  essential 
practices  organized  into  five  categories:  (1)  organization;  (2)  planning, 
scheduling  and  budgeting;  (3)  accounting;  (4)  analysis  and  management 
reports;  and  (6)  revisions  and  data  maintenance.  DOD  requires  that  all 
programs’  implementation  of  EVM  undergo  a  compliance  audit  against  the 
32  industry  practices.  In  addition,  DOD  policy  and  guidance48  state  that 
independent  surveillance  of  EVM  should  occur  over  the  life  of  the  program 
to  guarantee  the  validity  of  the  performance  data  and  ensure  that  EVM  is 
being  used  effectively  to  manage  cost,  schedule,  and  technical 
performance. 

On  Navy  ERP,  compliance  with  the  32  accepted  industry  practices  has  not 
been  verified,  and  surveillance  of  EVM  by  an  independent  entity  has  not 
occurred.  Therefore,  the  program  does  not  have  the  required  basis  for 
ensuring  that  EVM  is  being  effectively  implemented  on  Navy  ERP. 
According  to  program  officials,  surveillance  was  performed  by  NAVAIR 
for  Release  1.0.  However,  NAVAIR  officials  said  that  they  did  not  perform 
such  surveillance  because  they  did  not  receive  the  Release  1.0  cost 
performance  data  needed  to  do  so.  Program  officials  also  stated  that 
DON’s  Center  for  Earned  Value  Management49  has  conducted  an  initial 
assessment  of  their  EVM  management  system,  and  that  they  intend  to  have 
the  Center  perform  surveillance.  However,  they  did  not  have  a  plan  for 


4 ‘American  National  Standards  Institute  (ANSI)/Electronic  Industries  Alliance  (ELA) 
Standard,  EVM  Systems  Standard,  ANSI/ELA-748-A-1998  (R2002),  approved  May  19,  1998; 
revised  January  2002. 

48Defense  Contract  Management  Agency,  Department  of  Defense  Earned  Value 
Management  Implementation  Guide  (Washington,  D.C.:  October  2006).  See  also  DOD 
Memorandum:  Revision  to  DOD  Earned  Value  Manaqeynent  Policy  (Washington,  D.C.: 
Mar.  7,  2005). 

49DON  established  the  Center  for  Earned  Value  Management  in  April  2007  to,  among  other 
things,  work  with  program  offices  to  improve  the  accuracy  of  EVM  data  and  to  provide 
independent  assistance  to  program  managers  when  requested. 
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Estimated  Schedule  Baseline 
Was  Not  Derived  in 
Accordance  with  All  Key 
Schedule  Estimating  Practices 


accomplishing  this.  Until  compliance  with  the  standards  is  verified  and 
continuous  surveillance  occurs,  and  deviations  are  addressed,  the  program 
will  likely  continue  to  experience  cost  overruns  and  schedule  delays. 

The  success  of  any  program  depends  in  part  on  having  a  reliable  schedule 
of  when  the  program’s  work  activities  will  occur,  how  long  they  will  take, 
and  how  they  are  related  to  one  another.  As  such,  the  schedule  not  only 
provides  a  road  map  for  the  systematic  execution  of  a  program  but  also 
provides  the  means  by  which  to  estimate  costs,  gauge  progress,  identify 
and  address  potential  problems,  and  promote  accountability.  Our  research 
has  identified  nine  practices  associated  with  effective  schedule 
estimating.50  These  practices  are  (1)  capturing  key  activities,  (2) 
sequencing  key  activities,  (3)  establishing  the  duration  of  key  activities,  (4) 
assigning  resources  to  key  activities,  (5)  integrating  key  activities 
horizontally  and  vertically,  (6)  establishing  the  critical  path  for  key 
activities,  (7)  identifying  “float  time”51  between  key  activities,  (8) 
distributing  reserves  to  high-risk  activities,  and  (9)  performing  a  schedule 
risk  analysis. 

The  program’s  estimated  schedule  was  developed  using  some  of  these 
practices,  but  several  key  practices  were  not  fully  employed  that  are 
fundamental  to  having  a  schedule  that  provides  a  sufficiently  reliable  basis 
for  estimating  costs,  measuring  progress  and  forecasting  slippages.  On  the 
positive  side,  the  schedule  for  the  first  two  releases  captures  key  activities 
and  their  durations  and  is  integrated  horizontally  and  vertically,  meaning 
that  multiple  teams  executing  different  aspects  of  the  program  can 
effectively  work  to  the  same  master  schedule.  Moreover,  for  these  two 
releases,  the  program  has  established  float  time  between  key  activities  and 
distributed  schedule  reserve  to  high-risk  activities.  However,  the  program 
has  not  adequately  sequenced  and  assigned  resources  to  key  program 
activities.  Moreover,  the  estimated  schedule  for  the  first  increment  is  not 
grounded  in  an  integrated  master  schedule  of  ah  the  releases,  and  thus  the 
schedule  for  this  increment  does  not  reflect  the  program’s  critical  path  of 
work  that  must  be  performed  to  achieve  the  target  completion  date.  Also, 
it  does  not  reflect  the  results  of  a  schedule-risk  analysis  across  ah  three 
releases  with  schedule  reserve  allocated  to  high-risk  activities  because 
such  risks  were  not  examined.  See  table  6  for  the  results  of  our  analyses 
relative  to  each  of  the  nine  practices. 


50GAO-07-1 134SP. 

51Float  time  is  the  amount  of  time  an  activity  can  slip  before  affecting  the  critical  path. 
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Table  6:  Satisfaction  of  Schedule  Estimating  Key  Practices 


Practice 

Explanation 

Satisfied?8 

GAO  analysis 

Capturing 

key 

activities 

The  schedule  should  reflect  all  key  activities  (e.g.,  steps, 
events,  outcomes)  as  defined  in  the  program’s  work 
breakdown  structure,  to  include  activities  to  be  performed 
by  both  the  government  and  its  contractors. 

Yes 

The  program’s  estimated  schedules  for  the 
first  two  releases  reflect  both  government 
and  contractor  activities,  such  as 
development  and  testing  of  the  software 
components,  as  well  as  key  milestones  for 
measuring  progress. 

Sequencing 

key 

activities 

The  schedule  should  be  planned  so  that  it  can  meet  critical 
program  dates.  To  meet  this  objective,  key  activities  need 
to  be  logically  sequenced  in  the  order  that  they  are  to  be 
carried  out.  In  particular,  activities  that  must  be  finished 
prior  to  the  start  of  other  activities  (i.e.,  predecessor 
activities),  as  well  as  activities  that  cannot  begin  until  other 
activities  are  completed  (i.e.,  successor  activities)  should 
be  identified.  By  doing  so,  interdependencies  among 
activities  that  collectively  lead  to  the  accomplishment  of 
events  or  milestones  can  be  established  and  used  as  a 
basis  for  guiding  work  and  measuring  progress. 

Partially 

The  schedules  for  the  first  two  releases 
include  the  logical  sequencing  of  most,  but 
not  all,  activities.  For  example,  234  of  2,445 
activities  in  the  Release  1.1  schedule  were 
not  sequenced  properly.  By  not  identifying 
the  correct  interdependencies  and  properly 
sequencing  key  activities,  the  schedule  may 
not  facilitate  the  meaningful  tracking  of 
progress. 

Establishing 
the  duration 
of  key 
activities 

The  schedule  should  realistically  reflect  how  long  each 
activity  will  take  to  execute.  In  determining  the  duration  of 
each  activity,  the  same  rationale,  historical  data,  and 
assumptions  used  for  cost  estimating  should  be  used  for 
schedule  estimating.  Further,  these  durations  should  be  as 
short  as  possible,  and  they  should  have  specific  start  and 
end  dates.  Excessively  long  periods  needed  to  execute  an 
activity  should  prompt  further  decomposition  of  the  activity 
so  that  shorter  execution  durations  will  result. 

Yes 

The  schedules  for  the  first  two  releases 
establish  realistic  durations  of  key  activities. 
For  example,  the  schedule  for  Release  1 .1  is 
based  on  historical  data  from  Release  1 .0, 
which  provides  a  level  of  confidence  that  the 
durations  are  reasonable. 

Assigning 
resources 
to  key 
activities 

The  schedule  should  reflect  what  resources  (i.e.,  labor, 
material,  and  overhead)  are  needed  to  do  the  work, 
whether  all  required  resources  will  be  available  when  they 
are  needed,  and  whether  any  funding  or  time  constraints 
exist. 

Partially 

The  schedules  for  the  first  two  releases 
include  the  allocation  of  resources,  such  as 
personnel,  to  activities,  but  it  does  not  reflect 
whether  all  resources  will  be  available  when 
they  are  needed  because  the  identified 
resources  are  shared  across  the  three 
releases.  Restated,  personnel  are  assigned 
to  activities  across  multiple  releases,  each  of 
which  is  managed  according  to  a  separate 
schedule.  Therefore,  if  one  of  the  schedules 
were  to  be  delayed,  the  other  schedules  that 
required  the  same  resources  would  likely 
also  be  delayed. 
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Practice 

Explanation 

Satisfied?" 

GAO  analysis 

Integrating 

key 

activities 

horizontally 

and 

vertically 

The  schedule  should  be  horizontally  integrated,  meaning 
that  it  should  link  the  products  and  outcomes  associated 
with  already  sequenced  activities.  These  links  are 
commonly  referred  to  as  “handoffs”  and  serve  to  verify  that 
activities  are  arranged  in  the  right  order  to  achieve 
aggregated  products  or  outcomes.  The  schedule  should 
also  be  vertically  integrated,  meaning  that  traceability 
exists  among  varying  levels  of  activities  and  supporting 
tasks  and  subtasks.  Such  mapping  or  alignment  among 
levels  enables  different  groups  to  work  to  the  same  master 
schedule. 

Yes 

The  schedules  for  the  first  two  releases  are 
both  horizontally  and  vertically  integrated, 
meaning  that  the  activities  across  the 
multiple  teams  are  arranged  in  the  right 
order  to  achieve  aggregated  products  or 
outcomes.  In  addition,  traceability  exists 
among  varying  levels  of  activities,  which 
allows  multiple  teams  (i.e.,  development, 
testing)  to  work  to  the  same  master 
schedule. 

Establishing 
the  critical 
path  for  key 
activities 

Using  scheduling  software,  the  critical  path — the  longest 
duration  path  through  the  sequenced  list  of  key  activities — 
should  be  identified.  The  establishment  of  a  program’s 
critical  path  is  necessary  for  examining  the  effects  of  any 
activity  slipping  along  this  path.  Potential  problems  that 
might  occur  along  or  near  the  critical  path  should  also  be 
identified  and  reflected  in  the  scheduling  of  the  time  for 
high-risk  activities  (see  next  practice). 

Partially 

While  the  program  has  established  the 
critical  path  for  the  first  two  releases 
separately,  it  has  not  done  so  for  the  entire 
first  increment.  This  is  because  the  program 
maintains  separate  schedules  for  each 
release,  and  in  doing  so,  cannot  identify  the 
longest  duration  of  tasks  through  the  entire 
first  increment.  Without  doing  so,  the  effects 
of  any  slippage  along  the  critical  path  on 
future  releases  cannot  be  determined. 

Identifying 
“float  time” 
between 
key 

activities 

The  schedule  should  identify  float  time — the  time  that  a 
predecessor  activity  can  slip  before  the  delay  affects 
successor  activities — so  that  schedule  flexibility  can  be 
determined.  As  a  general  rule,  activities  along  the  critical 
path  typically  have  the  least  amount  of  float  time. 

Yes 

The  schedules  for  the  first  two  releases 
identify  float  time  between  key  activities. 

Distributing 
reserves  to 
high-risk 
activities 

The  baseline  schedule  should  include  a  buffer  or  a  reserve 
of  extra  time.  Schedule  reserve  for  contingencies  should 
be  calculated  by  performing  a  schedule  risk  analysis  (see 
next  practice).  As  a  general  rule,  the  reserve  should  be 
applied  to  high-risk  activities,  which  are  typically  found 
along  the  critical  path. 

Partially 

The  schedule  allocates  reserve  for  high-risk 
activities  on  the  critical  path  for  Release  1 .0. 
However,  because  the  program  has  not 
established  a  critical  path  for  the  first 
increment,  it  cannot  allocate  reserve  for  the 
high-risk  activities  on  the  entire  program’s 
critical  path. 

Schedule 

risk 

analysis 
should  be 
performed 

A  schedule  risk  analysis  should  be  performed  using 
statistical  techniques  to  predict  the  level  of  confidence  in 
meeting  a  program’s  completion  date.  This  analysis 
focuses  not  only  on  critical  path  activities  but  also  on 
activities  near  the  critical  path,  since  they  can  potentially 
affect  program  status. 

Partially 

A  schedule  risk  analysis  on  the  entire 
program  was  not  performed.  A  schedule  risk 
analysis  has  been  done  on  Release  1 .0,  and 
the  program  office  plans  to  do  one  for 

Release  1.1.  However,  without  analyzing  the 
risks  associated  with  the  program’s  entire 
schedule,  the  program  cannot  determine  the 
level  of  confidence  in  meeting  the  program’s 
completion  date. 

Source:  GAO  analysis  of  DON  data. 


"“Yes”  means  that  the  program  provided  documentation  demonstrating  satisfaction  of  the  practice. 
“Partially”  means  that  the  program  provided  documentation  demonstrating  satisfaction  of  part  of  the 
practice.  “No”  means  that  the  program  has  yet  to  provide  documentation  demonstrating  satisfaction  of 
the  practice. 
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Trends  in  EVM  Data  Show 
Pattern  of  Cost  Overruns  and 
Schedule  Slippages 


According  to  program  documentation,  they  have  plans  to  address  the 
logical  sequencing  of  activities  (to  ensure  that  it  reflects  how  work  is  to  be 
performed),  but  program  officials  stated  that  they  do  not  plan  to  combine 
all  three  releases  into  a  single  integrated  master  schedule  for  the  entire 
first  increment  of  the  program  because  doing  so  would  produce  an  overly 
complex  and  nonexecutable  schedule  involving  as  many  as  15,000 
activities.  However,  our  research  of  and  experience  in  evaluating  major 
programs’  use  of  EVM  and  integrated  master  schedules  show  that  while 
large,  complex  programs  necessitate  schedules  involving  thousands  of 
activities,  successful  programs  ensure  that  their  schedules  integrate  these 
activities.  In  our  view,  not  adequately  performing  these  practices  does  not 
allow  the  program  to  effectively  assign  resources,  identify  the  critical  path, 
and  perform  a  schedule  risk  analysis  that  would  allow  it  to  understand, 
disclose,  and  compensate  for  its  schedule  risks.  This  means  that  the 
program  is  not  well-positioned  to  understand  progress  and  forecast  its 
impact.  To  illustrate,  the  program  recently  experienced  delays  in 
deploying  its  first  release  at  NAVAIR,  which  according  to  a  recent 
operational  test  and  evaluation  report52  has  significantly  affected  the 
schedule’s  critical  path.  These  schedule  impacts  are  because  resources 
supporting  the  deployment  at  NAVAIR  began  to  shift  to  the  next  scheduled 
deployment  site  and  thus  are  no  longer  available  to  resolve  critical  issues 
at  NAVAIR.  Since  the  schedule  baseline  is  not  integrated  across  all 
releases,  the  impact  of  this  delay  on  other  releases,  and  thus  the  program 
as  a  whole,  cannot  be  readily  determined. 

Program  data  show  a  pattern  of  actual  cost  overruns  and  schedule  delays 
between  January  2007  and  May  2008.  Moreover,  our  analysis  of  the  data 
supports  a  most  likely  program  cost  growth  of  about  $152  million  to 
complete  all  three  releases. 


“Department  of  the  Navy,  Commander,  Operational  Test  and  Evaluation  Force,  Navy 
Enterprise  Resource  Planning  System  Initial  Operational  Test  and  Evaluation,  OT-C1 
Final  Report  to  the  Chief  of  Naval  Operations  (Norfolk,  VA:  June  13,  2008). 
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Differences  from  the  PMB  are  measured  in  both  cost  and  schedule 
variances.53  Positive  variances  indicate  that  activities  are  costing  less  or 
are  completed  ahead  of  schedule.  Negative  variances  indicate  that 
activities  are  costing  more  or  are  falling  behind  schedule.  These  cost  and 
schedule  variances  can  then  be  used  in  forecasting  the  cost  and  time 
needed  to  complete  the  program. 

Based  on  program-provided  data  for  the  first  increment  over  a  17-month 
period  ending  May  2008,  the  program  has  experienced  negative  cost 
variances.  Specifically,  while  these  cost  variances  have  fluctuated  during 
this  period,  they  have  consistently  been  negative.  (See  fig.  4.)  Moreover, 
our  analysis  of  the  cost  to  complete  just  the  budgeted  development  work 
(also  known  as  the  PMB)  for  all  three  releases,  which  is  about  $844 
million,54  will  be  exceeded  by  between  about  $102  million  and  $316  million, 
with  a  most  likely  overrun  of  about  $162  million.55  In  contrast,  the  program 
office  reports  that  the  overrun  at  completion  will  be  $55  million  but  has 
yet  to  provide  us  with  documentation  supporting  this  calculation. 
Moreover,  our  calculation  does  not  reflect  the  recent  problems  discovered 
during  the  operational  test  and  evaluation  at  NAVAIR  and  thus  the  overrun 
is  likely  to  be  higher. 


53Cost  variances  compare  the  earned  value  of  the  completed  work  with  the  actual  cost  of 
the  work  performed.  For  example,  if  a  contractor  completed  $5  million  worth  of  work,  and 
the  work  actually  cost  $6.7  million,  there  would  be  a  negative  $1.7  million  cost  variance. 
Schedule  variances  are  also  measured  in  dollars,  but  they  compare  the  earned  value  of  the 
work  actually  completed  as  of  a  point  in  time  to  the  value  of  work  that  was  expected  to  be 
completed.  For  example,  if  a  contractor  completed  $5  million  worth  of  work  at  the  end  of 
the  month  but  was  budgeted  to  complete  $10  million  worth  of  work,  there  would  be  a 
negative  $5  million  schedule  variance. 

54This  figure  is  the  total  estimated  budget  for  the  program.  It  represents  the  cumulative 
value  of  the  budgeted  cost  of  work  scheduled  over  the  life  of  the  program. 

55To  make  these  assessments,  we  applied  earned  value  analysis  techniques  to  data  from  the 
program’s  contract  performance  reports.  These  techniques  compare  budget  versus  actual 
costs  versus  project  status  in  dollar  amounts.  For  our  analysis,  we  used  standard  earned 
value  formulas  to  calculate  cost  and  schedule  variance  and  forecast  the  range  of  cost 
overrun  at  completion. 


Page  41 


GAO-08-896  DOD  Business  Systems  Modernization 


Figure  4:  Cumulative  Cost  Variance  for  Navy  ERP  over  a  17-Month  Period 
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Source:  GAO  analysis  based  on  Navy  ERP  data. 


During  this  same  17-month  period,  the  program  has  experienced  negative 
schedule  variances  and,  since  January  2008,  they  have  almost  doubled 
each  month.  Further,  as  of  May  2008,  the  program  had  not  completed 
about  $24  million  in  scheduled  work.  (See  fig.  5.)  An  inability  to  meet 
schedule  performance  is  a  frequent  indication  of  future  cost  increases,  as 
more  spending  is  often  necessary  to  resolve  schedule  delays. 
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Figure  5:  Cumulative  Schedule  Variance  of  the  Navy  ERP  Program  over  a  17-Month  Period 
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Source:  GAO  analysis  based  on  Navy  ERP  data. 


Because  the  program  office  has  not  performed  important  reliability 
checks,  such  as  EVM  validation  and  integrated  baseline  reviews,  as 
discussed  above,  we  cannot  be  certain  that  the  PMB  is  reliable  (i.e., 
reflects  all  the  work  to  be  done  and  has  identified  all  the  risks).  As  a  result, 
the  overrun  that  we  are  forecasting  could  be  higher. 

By  not  executing  basic  EVM  practices,  the  program  has  and  will  likely 
continue  to  experience  cost  and  schedule  shortfalls.  Until  the  program 
office  implements  these  important  EVM  practices,  it  will  likely  not  be  able 
to  track  actual  program  costs  and  schedules  close  to  estimates. 
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Important  Requirements 
Management  Activity  Has 
Been  Effectively 
Implemented 


Well-defined  and  managed  requirements  are  recognized  by  DOD  guidance 
as  essential,  and  they  can  be  viewed  as  a  cornerstone  of  effective  system 
acquisition.  One  aspect  of  effective  requirements  management  is 
requirements  traceability.56  By  tracing  requirements  both  backward  from 
system  requirements  to  higher  level  business  or  operational  requirements 
and  forward  to  system  design  specifications  and  test  plans,  the  chances  of 
the  deployed  product  satisfying  requirements  is  increased,  and  the  ability 
to  understand  the  impact  of  any  requirement  changes  and  thus  make 
informed  decisions  about  such  changes,  is  enhanced. 


The  program  office  is  effectively  implementing  requirements  traceability 
for  its  1,733  Release  1.0  system  requirements.  To  verify  this  traceability, 
we  randomly  selected  and  analyzed  60  of  the  1,733  system  requirements 
and  confirmed  that  58  of  the  60  were  traceable  both  backward  to  higher 
level  requirements  and  forward  to  design  specifications  and  test  results. 
The  remaining  2  had  been  allocated  to  the  other  releases,  and  thus  we  also 
confirmed  the  program’s  ability  to  maintain  traceability  between  product 
releases.  In  doing  so,  the  program  utilized  a  tool  called  DOORS,  which  if 
implemented  properly,  allows  each  requirement  to  be  linked  from  its  most 
conceptual  definition  to  its  most  detailed  definition,  as  well  as  to  design 
specifications  and  test  cases.  In  effect,  the  tool  maintains  the  linkages 
among  requirement  documents,  design  documents,  and  test  cases  even  if 
requirements  change. 


If  DON  continues  to  effectively  implement  requirements  traceability,  it  will 
increase  the  chances  that  system  requirements  will  be  met  by  the  deployed 
system. 


A  Risk  Management 
Process  Has  Been  Defined, 
but  All  Risks  Have  Not 
Been  Effectively  Mitigated 


Proactively  managing  program  risks  is  a  key  acquisition  management 
control  and,  if  defined  and  implemented  properly,  it  can  increase  the 
chances  of  programs  delivering  promised  capabilities  and  benefits  on  time 
and  within  budget.  To  the  program  office’s  credit,  it  has  defined  a  risk 
management  process  that  meets  relevant  guidance.  However,  it  has  not 
effectively  implemented  the  process  for  all  identified  risks.  As  a  result, 
these  risks  have  not  been  proactively  mitigated  and  either  have 


56DOD ,  Defense  Acquisition  Guidebook,  Version  1.0  (Oct.  17,  2004).  Software  Engineering 
Institute,  Software  Acquisition  Capability  Maturity  Model®  version  1.03,  CMU/SEI-2002- 
TR-010  (Pittsburgh,  PA:  March  2002). 
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contributed  to  cost  and  schedule  shortfalls,  or  could  potentially  contribute 
to  such  shortfalls. 

DOD  acquisition  management  guidance,  as  well  as  other  relevant  guidance 
advocates  identifying  facts  and  circumstances  that  can  increase  the 
probability  of  an  acquisition’s  failing  to  meet  cost,  schedule,  and 
performance  commitments  and  then  taking  steps  to  reduce  the  probability 
of  their  occurrence  and  impact.57  In  brief,  effective  risk  management 
consists  of:  (1)  establishing  a  written  plan  for  managing  risks;  (2) 
designating  responsibility  for  risk  management  activities;  (3)  encouraging 
program-wide  participation  in  the  identification  and  mitigation  of  risks;  (4) 
defining  and  implementing  a  process  that  provides  for  the  identification, 
analysis,  and  mitigation  of  risks;  and  (5)  examining  the  status  of  identified 
risks  in  program  milestone  reviews. 

The  program  office  has  developed  a  written  plan  for  managing  risks  and 
established  a  process  that  together  provide  for  the  above-cited  risk 
management  practices.  Moreover,  it  has  largely  followed  its  plan  and 
process  as  per  the  following  examples: 

•  The  program  manager  has  been  assigned  overall  responsibility  for 
managing  risks  and  serves  as  the  chair  of  the  risk  management  board.58 
Also,  a  functional  team  lead  (i.e.,  subject  matter  expert)  is  assigned 
responsibility  for  analyzing  and  mitigating  each  identified  risk. 

•  Program-wide  participation  in  the  identification,  analysis,  and  mitigation 
of  risks  is  encouraged.  Specifically,  a  manager  for  each  release  is 
responsible  for  providing  risk  management  guidance  to  the  staff,  which 
includes  staff  identification  and  analysis  of  risks.  Also,  according  to  the 
program  office’s  risk  management  plan,  all  program  personnel  can  submit 
a  risk  for  approval.  In  addition,  stakeholders  participate  in  risk 
management  activities  during  acquisition  milestone  reviews. 


’‘Department  of  Defense,  Risk  Management  Guide  for  DOD  Acquisition,  6th  Edition, 
Version  1.0.,  http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-version.pdf 
(accessed  March  13,  2008)  and  Software  Engineering  Institute,  CMMI  for  Acquisition, 
Version  1.2,  CMU/SEI-2007-TR-017  (Pittsburgh,  PA:  November  2007). 

58The  risk  management  board  oversees  risk  management  activities  by  assigning  risk 
owners,  approving  and  directing  resources  to  facilitate  risk  handling  strategies,  and 
reviewing  risk  handling  status. 
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•  The  program  office  has  identified  and  categorized  individual  risks.  As  of 
June  2008,  the  risk  database  contained  15  active  risks — 3  high,  8  medium, 
and  4  low.59 

•  Program  risks  are  considered  during  program  milestone  reviews.  For 
example,  during  the  program’s  critical  design  review,  which  is  a  key  event 
of  the  system  development  and  demonstration  phase,  key  risks  regarding 
implementing  new  business  processes  and  legacy  system  changes  were 
discussed.  Furthermore,  the  program  manager  receives  a  monthly  risk 
report  that  describes  the  status  of  program  risks. 

However,  the  program  office  has  not  consistently  followed  other  aspects  of  its 
process.  In  particular,  it  has  not  effectively  implemented  steps  for  mitigating  the 
risks  associated  with  (1)  converting  data  from  NAVAIR’s  legacy  systems  to  nm 
on  Navy  ERP  and  (2)  positioning  NAVAIR  for  adopting  the  new  business 
processes  embedded  in  Navy  ERP.  As  we  have  previously  reported,  it  is 
important  for  organizations  that  are  to  operate  and  use  commercial  off-the-shelf 
software  products,  such  as  Navy  ERP,  to  proactively  manage  and  position 
themselves  for  the  organizational  impact  of  introducing  functionality 
embedded  in  the  commercial  products.  If  they  do  not,  the  organization’s 
performance  will  suffer.60 

To  the  program  office’s  credit,  it  identified  numerous  risks  associated  with 
data  conversion  and  organizational  change  management  and  developed 
and  implemented  strategies  that  were  intended  to  mitigate  these  risks. 
However,  it  closed  these  risks  even  though  they  were  never  effectively 
mitigated,  as  evidenced  by  the  results  of  recently  completed  DON 
operational  test  and  evaluation.  According  to  the  June  2008  operational 
test  and  evaluation  report  for  NAVAIR,  significant  problems  relating  to 
both  legacy  system  data  conversion  and  adoption  of  new  business 
processes  were  experienced.  The  report  states  that  these  problems  have 
contributed  to  increases  in  the  costs  to  operate  the  system,  including 


59Risk  levels  of  high,  medium  or  low  are  assigned  using  quantitative  measurements  of  the 
probability  of  the  risk  occurring  and  the  potential  impact  to  the  program’s  cost,  schedule, 
and  performance.  Based  on  that  assessment,  a  risk  level  is  assigned  to  represent  the  risk’s 
significance. 

60See,  for  example,  GAO,  Office  of  Personnel  Management:  Retirement  Systems 
Modernization  Program  Faces  Numerous  Challenges,  GAO-05-237  (Washington,  D.C.: 

Feb.  28,  2005);  and  GAO,  Information  Technology.  Customs  Automated  Commercial 
Environment  Program  Progressing,  but  Need  for  Management  Improvements  Continues, 
GAO-05-267  (Washington,  D.C.:  Mar.  14,  2005). 
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unexpected  manual  effort.  It  further  states  that  these  problems  have 
rendered  the  deployed  version  not  operationally  effective  and  that 
deployment  of  the  system  to  other  sites  should  not  occur  until  the  change 
management  process  has  been  analyzed  and  improved.  It  also  attributed 
the  realization  of  the  problems  to  the  program  office  and  NAVAIR  not 
having  adequately  engaged  and  communicated  early  with  each  other  to 
coordinate  and  resolve  differences  in  organizational  perspectives  and 
priorities  and  provide  intensive  pre-deployment  preparation  and  training. 
Program  officials  acknowledged  these  shortcomings  and  attributed  them 
to  their  limited  authority  over  the  commands.  In  this  regard,  they  have 
previously  surfaced  these  risks  with  department  oversight  and  approval 
authorities,  but  actions  were  not  taken  by  these  authorities  that  ensured 
that  the  risks  were  being  effectively  mitigated. 

Beyond  not  effectively  mitigating  these  risks,  the  program  office  has  not 
ensured  that  all  risks  are  captured  in  the  risk  inventory.  For  example,  the 
inventory  does  not  include  the  risks  described  in  this  report  that  are 
associated  with  not  having  adequately  demonstrated  the  program’s 
alignment  to  the  federated  BEA  and  not  having  implemented  program- 
level  EVM  in  a  manner  that  reflects  industry  practices.  This  means  that 
these  risks  are  not  being  disclosed  or  mitigated. 

By  not  effectively  addressing  all  risks  associated  with  the  program,  these 
risks  can  and  have  become  problems  that  contribute  to  cost  and  schedule 
shortfalls.  Until  all  significant  risks  are  proactively  addressed,  to  include 
ensuring  that  all  associated  mitigation  steps  are  implemented  and  that  they 
accomplished  their  intended  purpose,  the  program  will  likely  experience 
further  problems  at  subsequent  deployment  sites. 


Conclusions 


DOD’s  success  in  delivering  large-scale  business  systems,  such  as  Navy 
ERP,  is  in  large  part  determined  by  the  extent  to  which  it  employs  the  kind 
of  rigorous  and  disciplined  IT  management  controls  that  are  reflected  in 
department  policies  and  related  guidance.  While  implementing  these 
controls  does  not  guarantee  a  successful  program,  it  does  minimize  a 
program’s  exposure  to  risk  and  thus  the  likelihood  that  it  will  fall  short  of 
expectations.  In  the  case  of  Navy  ERP,  living  up  to  expectations  is 
important  because  the  program  is  large,  complex,  and  critical  to 
addressing  the  department’s  long-standing  problems  related  to  financial 
transparency  and  asset  visibility. 

The  effectiveness  to  which  key  IT  management  controls  have  been 
implemented  in  Navy  ERP  varies,  with  one  control  and  several  aspects  of 
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others  being  effectively  implemented,  and  others  less  so.  Moreover,  those 
controls  that  have  not  been  effectively  implemented  have,  in  part, 
contributed  to  the  sizable  cost  and  schedule  shortfalls  experienced  to  date 
on  the  program.  Unless  this  changes,  more  shortfalls  can  be  expected. 

While  the  program  office  is  primarily  responsible  for  ensuring  that  effective  IT 
management  controls  are  implemented,  other  oversight  and  stakeholder 
organizations  share  responsibility.  For  example,  even  though  the  program  has 
not  demonstrated  its  alignment  with  the  federated  BEA,  it  nevertheless 
followed  established  DOD  architecture  compliance  guidance  and  used  the 
related  compliance  assessment  tool  in  assessing  and  asserting  its  compliance. 
The  root  cause  for  not  demonstrating  compliance  thus  is  not  traceable  to  the 
program  office  but  rather  is  due  to,  among  other  tilings,  the  limitations  of  the 
compliance  guidance  and  tool,  and  the  program’s  oversight  entities  not 
validating  the  compliance  assessment  and  assertion.  Also,  the  reason  that  the 
program’s  cost  estimate  was  not  informed  by  the  cost  experiences  of  other 
programs  of  the  same  size  and  scope  is  because  DOD  does  not  have  a  standard 
ERP  cost  element  structure  and  has  not  maintained  a  historical  database  of 
costs  for  like  programs  to  use.  In  contrast,  effective  implementation  of  other 
management  controls,  such  as  implementing  EVM,  requirements  traceability, 
and  risk  management  is  the  responsibility  of  the  program  office. 

All  told,  addressing  the  management  control  weaknesses  requires  the 
combined  efforts  of  the  various  organizations  that  share  responsibility  for 
managing  and  overseeing  the  program.  By  doing  so,  the  department  can 
better  assure  itself  that  Navy  ERP  will  optimally  support  its  performance 
goals  and  will  deliver  promised  capabilities  and  benefits  on  time  and 
within  budget. 

Because  we  recently  completed  work  that  more  broadly  addresses  the 
above  cited  architectural  alignment  and  comparable  program  cost  data 
limitations,  we  are  not  making  recommendations  in  this  report  for 
addressing  them. 


Recommendations 
Executive  Action 


To  strengthen  Navy  ERP  management  control  and  better  provide  for  the 
program’s  success,  we  are  making  the  following  recommendations: 


•  To  improve  the  reliability  of  Navy  ERP  benefit  estimates  and  cost 
estimates,  we  recommend  that  the  Secretary  of  Defense  direct  the 
Secretary  of  the  Navy,  through  the  appropriate  chain  of  command,  to 
ensure  that  future  Navy  ERP  estimates  include  uncertainty  analyses  of 
estimated  benefits,  reflect  the  risks  associated  with  not  having  cost  data 
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for  comparable  ERP  programs,  and  are  otherwise  derived  in  full 
accordance  with  the  other  key  estimating  practices,  and  economic  analysis 
practices  discussed  in  this  report. 

•  To  enhance  Navy  ERP’s  use  of  EVM,  we  recommend  that  the  Secretary  of 
Defense  direct  the  Secretary  of  the  Navy,  through  the  appropriate  chain  of 
command,  to  ensure  that  (1)  an  integrated  baseline  review  on  the  last  two 
releases  of  the  first  increment  is  conducted,  (2)  compliance  against  the  32 
accepted  industry  EVM  practices  is  verified,  and  (3)  a  plan  to  have  an 
independent  organization  perform  surveillance  of  the  program’s  EVM 
system  is  developed  and  implemented. 

•  To  increase  the  quality  of  the  program’s  integrated  master  schedule,  we 
recommend  that  the  Secretary  of  Defense  direct  the  Secretary  of  the  Navy, 
through  the  appropriate  chain  of  command,  to  ensure  that  the  schedule  (1) 
includes  the  logical  sequencing  of  all  activities,  (2)  reflects  whether  all 
required  resources  will  be  available  when  needed,  (3)  defines  a  critical 
path  that  integrates  all  three  releases,  (4)  allocates  reserve  for  the  high- 
risk  activities  on  the  entire  program’s  critical  path,  and  (5)  incorporates 
the  results  of  a  schedule  risk  analysis  for  all  three  releases  and 
recalculates  program  cost  and  schedule  variances  to  more  accurately 
determine  a  most  likely  cost  and  schedule  overrun. 

•  To  improve  Navy  ERP’s  management  of  program  risks,  we  recommend 
that  the  Secretary  of  Defense  direct  the  Secretary  of  the  Navy,  through  the 
appropriate  chain  of  command,  to  ensure  that  (1)  the  plans  for  mitigating 
the  risks  associated  with  converting  data  from  legacy  systems  to  Navy 
ERP  and  positioning  the  commands  for  adopting  the  new  business 
processes  embedded  in  the  Navy  ERP  are  re-evaluated  in  light  of  the 
recent  experience  with  NAVAIR  and  adjusted  accordingly,  (2)  the  status 
and  results  of  these  and  other  mitigation  plans’  implementation  are 
periodically  reported  to  program  oversight  and  approval  authorities,  (3) 
these  authorities  ensure  that  those  entities  responsible  for  implementing 
these  strategies  are  held  accountable  for  doing  so,  and  (4)  each  of  the  risks 
discussed  in  this  report  are  included  in  the  program’s  inventory  of  active 
risks  and  managed  accordingly. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  signed  by  the  Deputy  Under 
Secretary  of  Defense  (Business  Transformation)  and  reprinted  in  appendix 
II,  DOD  stated  that  it  concurred  with  two  of  our  four  recommendations 
and  partially  concurred  with  the  remaining  two.  Further,  it  stated  that  it 
has  taken  steps  to  address  some  of  our  recommendations,  adding  that  it  is 
committed  to  implementing  recommendations  that  contribute  to  the 


Page  49 


GAO-08-896  DOD  Business  Systems  Modernization 


program’s  success.  The  department’s  comments  relative  to  both  of  the 
recommendations  that  it  partially  concurred  with,  as  well  as  additional 
comments,  are  discussed  below. 

For  our  recommendation  associated  with  improving  the  program’s  benefit 
and  cost  estimates,  DOD  concurred  with  two  of  the  recommendation’s  three 
parts,  but  it  did  not  concur  with  one  part — ensuring  that  future  cost  estimates 
reflect  the  risk  of  not  having  cost  data  for  comparable  programs.  While 
acknowledging  that  the  program  had  limited  cost  data  from  comparable 
programs  on  which  to  base  its  cost  estimate,  DOD  stated  that  an  uncertainty 
analysis  had  been  applied  to  the  estimate  to  account  for  the  risk  associated 
with  not  having  such  data.  The  department  further  stated  that  actual 
experience  on  the  program  will  continue  to  be  used  to  refine  the  program’s 
cost  estimating  methodology.  While  we  support  DOD’s  stated  commitment  to 
using  actual  program  cost  experience  in  deriving  future  estimates,  we  do  not 
agree  that  the  latest  estimate  accounted  for  the  risk  of  not  having  cost  data 
from  comparable  programs.  We  examined  the  uncertainty  analysis  as  part  of 
our  review,  and  found  that  it  did  not  recognize  this  risk.  Moreover,  DOD’s 
comments  offered  no  new  evidence  to  the  contrary. 

For  our  recommendation  associated  with  improving  the  program’s 
schedule  estimating,  DOD  concurred  with  four  of  the  recommendation’s 
five  parts,  and  partially  concurred  with  one  part — ensuring  that  the 
schedule  defines  a  critical  path  that  integrates  all  releases.  In  taking  this 
position,  the  department  stated  that  a  critical  path  has  been  established  for 
each  release  rather  than  across  all  three  releases,  and  it  attributes  this  to 
the  size  and  complexity  of  the  program.  We  do  not  take  issue  with  either 
of  these  statements,  as  they  are  already  recognized  in  our  report.  However, 
DOD  offers  no  new  information  in  its  comments.  Further,  our  report  also 
recognizes  that  to  be  successful,  large  and  complex  programs  that  involve 
thousands  of  activities  need  to  ensure  that  their  schedules  integrate  these 
activities.  In  this  regard,  we  support  the  department’s  commitment  to 
explore  the  feasibility  of  implementing  this  part  of  our  recommendation. 

In  addition,  while  stating  that  it  concurred  with  all  parts  of  our 
recommendation  associated  with  improving  the  program’s  use  of  EVM, 
DOD  nevertheless  provided  additional  comments  as  justification  for 
having  not  conducted  an  integrated  baseline  review  on  Release  1.0. 
Specifically,  it  stated  that  when  it  rebaselined  this  release  in  December 
2006,  the  release’s  development  activities  were  essentially  complete  and 
the  release  was  in  the  latter  stages  of  testing.  Further,  it  stated  that  the 
risks  associated  with  the  Release  1.0  schedule  were  assessed  3  months 
after  this  rebaselining,  and  these  risks  were  successfully  mitigated.  To 
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support  this  statement,  it  said  that  Release  1.0  achieved  its  “Go-Live”  as 
scheduled  at  NAVAIR.  We  do  not  agree  with  these  comments  for  several 
reasons.  First,  at  the  time  of  the  rebaselining,  about  9  months  of  scheduled 
Release  1.0  development  remained,  and  thus  the  release  was  far  from 
complete.  Moreover,  the  significance  of  the  amount  of  work  that 
remained,  and  still  remains  today  on  Release  1.0  is  acknowledged  in 
DOD’s  own  comment  that  the  scheduled  integrated  baseline  review  for 
Release  1.1  will  also  include  remaining  Release  1.0  work.  Second,  the 
Release  1.0  contract  was  awarded  in  January  2006,  and  DOD’s  own 
guidance  requires  that  an  integrated  baseline  review  be  conducted  within 
6  months  of  a  contract’s  award.  Third,  although  DOD  states  that  the 
program  achieved  “Go-Live”  as  scheduled  on  October  1,  2007,  the  program 
achieved  initial  operational  capability  7  months  later  than  established  in 
the  December  2006  baseline. 

In  addition  to  these  comments,  the  department  also  described  actions 
under  way  or  planned  to  address  our  recommendations.  We  support  the 
actions  described,  as  they  are  consistent  with  the  intent  of  our 
recommendations.  If  fully  and  properly  implemented,  these  actions  will  go 
a  long  way  in  addressing  the  management  control  weaknesses  that  our 
recommendations  are  aimed  at  correcting. 


We  are  sending  copies  of  this  report  to  interested  congressional 
committees;  the  Director,  Office  of  Management  and  Budget;  the 
Congressional  Budget  Office;  the  Secretary  of  Defense;  and  the 
Department  of  Defense  Office  of  the  Inspector  General.  We  also  will  make 
copies  available  to  others  upon  request.  In  addition,  the  report  will  be 
available  at  no  charge  on  the  GAO  Web  site  at  http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact  me 
at  (202)  512-3439  or  hiter@gao.gov.  Contact  points  for  our  Offices  of 
Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  GAO  staff  who  made  major  contributions  to  this  report  are 
listed  in  appendix  III. 


Randolph  C.  Hite 

Director,  Information  Technology  Architecture 
and  Systems  Issues 
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Appendix  I:  Objective,  Scope,  and 
Methodology 


Our  objective  was  to  determine  whether  the  Department  of  the  Navy  is 
effectively  implementing  information  technology  management  controls  on 
the  Navy  Enterprise  Resource  Planning  (Navy  ERP)  program.  To 
accomplish  this,  we  focused  on  the  first  increment  of  Navy  ERP  and  the 
following  management  areas  (1)  architectural  alignment,  (2)  economic 
justification,  (3)  earned  value  management  (EVM),  (4)  requirements 
management,  and  (5)  risk  management. 

•  To  determine  whether  Navy  ERP  was  aligned  with  the  Department  of 
Defense’s  (DOD)  federated  business  enterprise  architecture  (BEA),  we 
reviewed  the  program’s  BEA  compliance  assessments  and  system 
architecture  products,  as  well  as  Versions  4.0,  4.1,  and  5.0  of  the  BEA,  and 
compared  them  with  the  BEA  compliance  requirements  described  in  the 
Fiscal  Year  2005  Defense  Authorization  Act1  and  DOD’s  BEA  compliance 
guidance,  and  we  evaluated  the  extent  to  which  the  compliance 
assessments  addressed  all  relevant  BEA  products.  We  also  determined  the 
extent  to  which  the  program-level  architecture  documentation  supported 
the  BEA  compliance  assessments.  We  obtained  documentation,  such  as 
the  BEA  compliance  assessments  from  the  Navy  ERP  and  Global  Combat 
Support  System — Marine  Corps  programs,  as  well  as  the  Air  Force’s 
Defense  Enterprise  Accounting  and  Management  System  and  Air  Force 
Expeditionary  Combat  Support  System  programs.  We  then  compared 
these  assessments  to  identify  potential  redundancies  or  opportunities  for 
reuse  and  determined  if  the  compliance  assessments  examined  duplication 
across  programs,  and  if  the  tool  that  supports  these  assessments  is  being 
used  to  identify  such  duplication.  In  doing  so,  we  interviewed  program 
officials  and  officials  from  the  Department  of  the  Navy,  Office  of  the  Chief 
Information  Officer  and  reviewed  recent  GAO  reports  to  determine  the 
extent  to  which  the  programs  were  assessed  for  compliance  against  the 
Department  of  the  Navy  enterprise  architecture.  We  also  interviewed 
program  officials  and  officials  from  the  Business  Transformation  Agency 
and  the  Department  of  the  Navy,  including  the  logistics  Functional  Area 
Manager,  and  obtained  guidance  documentation  from  these  officials  to 
determine  the  extent  to  which  the  compliance  assessments  were  subject 
to  oversight  or  validation. 

•  To  determine  whether  the  program  had  economically  justified  its 
investment  in  Navy  ERP,  we  reviewed  the  latest  economic  analysis  to 
determine  the  basis  for  the  cost  and  benefit  estimates.  This  included 


Donald  W.  Reagan  National  Defense  Authorization  Act  for  Fiscal  Year  2005,  Pub.  L. 
No.  108-375,  §  332  (2004)  (codified  at  10  U.S.C.  §§  186  and  2222). 
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evaluating  the  analysis  against  Office  of  Management  and  Budget  guidance 
and  GAO’s  Cost  Assessment  Guide.2  In  doing  so,  we  interviewed  cognizant 
program  officials,  including  the  program  manager  and  cost  analysis  team, 
regarding  their  respective  roles,  responsibilities,  and  actual  efforts  in 
developing  and/or  reviewing  the  economic  analysis.  We  also  interviewed 
officials  at  the  Office  of  Program  Analysis  and  Evaluation  and  Naval 
Center  for  Cost  Analysis  as  to  their  respective  roles,  responsibilities,  and 
actual  efforts  in  developing  and/or  reviewing  the  economic  analysis.  We 
did  not  verify  the  validity  of  the  source  data  used  to  calculate  estimated 
benefits,  such  as  those  data  used  to  determine  the  yearly  costs  associated 
with  legacy  systems  planned  for  retirement. 

•  To  determine  the  extent  to  which  the  program  had  effectively 

implemented  EVM,  we  reviewed  relevant  documentation,  such  as  contract 
performance  reports,  acquisition  program  baselines,  performance 
measurement  baseline,  and  schedule  estimates  and  compared  them  with 
DOD  policies  and  guidance.3  To  identify  trends  that  could  affect  the 
program  baseline  in  the  future,  we  assessed  cost  and  schedule 
performance  and,  in  doing  so,  we  applied  earned  value  analysis 
techniques4  to  data  from  contract  performance  reports.  We  compared  the 
cost  of  work  completed  with  the  budgeted  costs  for  scheduled  work  over 
a  17-month  period,  from  January  2007  to  May  2008,  to  show  trends  in  cost 
and  schedule  performance.  We  also  used  data  from  the  reports  to  estimate 
the  likely  costs  at  completion  of  the  program  through  established  earned 
value  formulas.  This  resulted  in  three  different  values,  with  the  middle 
value  being  the  most  likely.  We  checked  EVM  data  to  see  if  there  were  any 
mathematical  errors  or  inconsistencies  that  would  lead  to  the  data  being 
unreliable.  We  interviewed  cognizant  officials  from  the  Naval  Air  Systems 
Command  and  program  officials  to  determine  whether  the  program  had 
conducted  an  integrated  baseline  review,  whether  the  EVM  system  had 


2Office  of  Management  and  Budget,  Circular  No.  A-94:  Guidelines  and  Discount  Rates  for 
Benefit-Cost  Analysis  of  Federal  Program  (Oct.  29,  1992),  GAO-07-1 134SP. 

3Defense  Contract  Management  Agency,  Department  of  Defense  Earned  Value 
Management  Implementation  Guide  (Washington,  D.C.:  Apr.  7,  2005).  See  also  DOD 
Memorandum:  Revision  to  DOD  Earned  Value  Management  Policy  (Washington,  D.C.: 
Mar.  7,  2005). 

4The  earned  value  concept  is  applied  as  a  means  of  placing  a  dollar  value  on  project  status. 
The  techniques  we  used  compared  the  program’s  budget  to  actual  costs  and  project  status 
in  dollar  amounts.  We  used  standard  earned  value  formulas  to  calculate  cost  and  schedule 
variance  and  forecast  the  range  of  cost  overrun  at  program  completion. 
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been  validated  against  industry  guidelines,5  and  to  better  understand  the 
anomalies  in  the  EVM  data  and  determine  what  outside  surveillance  was 
being  done  to  ensure  that  the  industry  standards  are  being  met.  We  also 
reviewed  the  program’s  schedule  estimates  and  compared  them  with 
relevant  best  practices6  to  determine  the  extent  to  which  they  reflect  key 
estimating  practices  that  are  fundamental  to  having  a  reliable  schedule.  In 
doing  so,  we  interviewed  cognizant  program  officials  to  discuss  their  use 
of  best  practices  in  creating  the  program’s  current  schedule. 

•  To  determine  the  extent  to  which  the  program  has  effectively  implemented 
requirements  management,  we  reviewed  relevant  program  documentation, 
such  as  the  program  management  plan  and  baseline  list  of  requirements. 

To  determine  the  extent  to  which  the  program  has  maintained  traceability 
backward  to  high-level  business  operation  requirements  and  system 
requirements,  and  forward  to  system  design  specifications,  and  test  plans, 
we  randomly  selected  60  program  requirements  and  traced  them  both 
backward  and  forward.  This  sample  was  designed  with  a  6  percent 
tolerable  error  rate  at  the  95  percent  level  of  confidence  so  that,  if  we 
found  0  problems  in  our  sample,  we  could  conclude  statistically  that  the 
error  rate  was  less  than  5  percent.  In  addition,  we  interviewed  program 
officials  involved  in  the  requirements  management  process  to  discuss  their 
roles  and  responsibilities  for  managing  requirements. 

•  To  determine  the  extent  to  which  the  program  implemented  risk 
management,  we  reviewed  relevant  risk  management  documentation,  such 
as  the  program’s  risk  management  plan  and  risk  database  reports 
demonstrating  the  status  of  the  program’s  major  risks  and  compared  the 
program  office’s  activities  with  DOD  acquisition  management  guidance 
and  related  industry  practices.7  We  also  reviewed  the  program’s  mitigation 
process  with  respect  to  key  risks  to  determine  the  extent  to  which  these 
risks  were  effectively  managed.  In  doing  so,  we  interviewed  cognizant 
program  officials,  such  as  the  program  manager  and  risk  manager,  to 
discuss  their  roles  and  responsibilities  and  obtain  clarification  on  the 


’American  National  Standards  Institute  (ANSI)  /Electronic  Industries  Alliance  (EIA) 

EVM  Systems  Standard,  ANSI/EIA-748-A-1998  (R2002),  approved  May  19,  1998;  revised 
January  2002. 

6GAO-07-1134SP. 

Department  of  Defense,  Risk  Management  Guide  for  DOD  Acquisition,  6th  Edition, 
Version  1.0.,  http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-version.pdf 
(accessed  March  13,  2008)  and  Software  Engineering  Institute,  CMMI  for  Acquisition, 
Version  1.2,  CMU/SEI-2007-TR-017  (Pittsburgh,  PA:  November  2007). 
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program’s  approach  to  managing  risks  associated  with  acquiring  and 
implementing  Navy  ERP. 

We  conducted  this  performance  audit  at  DOD  offices  in  the  Washington, 
D.C.,  metropolitan  area  and  Annapolis,  Md.,  from  June  2007  to  September 
2008,  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objective.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objective. 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301*3000 


ACQUISITION 
TECHNOLOGY 
AND  LOGISTICS 

AUG  1  9 

Mr.  Randolph  C.  Hite 

Director,  Information  Technology  Architecture  and  Systems  Issues 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Mr.  Hite: 


This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report  GAO- 
08-896,  “DEFENSE  BUSINESS  SYSTEMS  MODERNIZATION:  Important 
Management  Controls  Being  Implemented  on  Major  Navy  Program,  But  Improvements 
Needed  in  Key  Areas,”  dated  July  18,  2008  (GAO  Code  310659).  Detailed  comments  on 
the  report  recommendations  are  enclosed. 

The  Department  concurred  with  two  recommendations  and  partially  concurred 
with  the  remaining  two.  The  Department  has  already  taken  steps  to  address  some  of 
GAO’s  recommendations  and  the  Navy  Enterprise  Resource  Planning  (ERP)  program 
office  is  committed  to  implementing  recommendations  that  will  contribute  to  the 
program's  success. 

We  appreciate  the  GAO’s  input  on  the  Department' s  progress  with  its  business 
transformation  efforts  and  continue  to  value  our  partnership. 


Sincerely. 


Pai l\  A.  Brinkley 

Deputy  Under  Secretary  of  Defense 
(Business  Transformation) 


Enclosure: 
As  stated 
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GAO  DRAFT  REPORT  DATED  JULY  18,  2008 
GAO-08-896  (GAO  CODE  310659) 


“DOD  BUSINESS  SYSTEMS  MODERNIZATION:  IMPORTANT 
MANAGEMENT  CONTROLS  BEING  IMPLEMENTED  ON  MAJOR 
NAVY  PROGRAM,  BUT  IMPROVEMENTS  NEEDED  IN  KEY  AREAS” 


DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  1 :  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy,  through  the  appropriate  chain  of  command,  to  ensure  that 
future  Navy  Enterprise  Resource  Planning  (ERP)  estimates  include  uncertainty  analyses 
of  estimated  benefits,  reflect  the  risk  associated  with  not  having  cost  data  for  comparable 
ERP  programs,  and  are  otherwise  derived  in  full  accordance  with  the  other  key  estimating 
practices,  and  economic  analysis  practices  discussed  in  this  report.  (Page  52/GAO  Draft 
Report) 

DOD  RESPONSE:  Partially  Concur.  Future  Navy  ERP  estimates  will  include 
uncertainty  analyses  of  estimated  benefits  and  be  derived  in  full  accordance  with  key 
estimating  and  economic  analysis  practices.  In  future  estimates,  uncertainty  surrounding 
the  assumptions  driving  the  benefit  estimate  will  have  probability  distributions  applied. 
The  point  estimate  reported  will  be  generated  using  the  expected  values  of  the  probability 
distributions. 

The  Department  does  not  concur  with  the  second  element  of  Recommendation  1,  that 
future  Navy  ERP  estimates  reflect  the  risk  associated  with  not  having  cost  data  for 
comparable  ERP  programs.  At  Milestone  A/B,  the  program  estimate  was  based  on 
commercial  best  practices  and  standards  used  to  implement  System  Application  Program 
(SAP)  functionality  as  well  as  engineering  inputs  and  expertise  developed  during  the 
Navy’s  pilot  ERP  programs.  This  approach  was  considered  the  best  approach  given  the 
limited  comparable  enterprise  program  system  implementation  data  currently  available  in 
DoD.  Furthermore,  the  industry  standards  and  the  functional  scope  definition  had 
uncertainty  analysis  applied  to  account  for  the  risk  associated  with  not  having  comparable 
DoD  programmatic  data.  As  the  program  began  execution,  the  methodology  was  and 
continues  to  be  refined  as  actual  experience  is  gained  by  the  Navy  ERP  program. 

RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy,  through  the  appropriate  chain  of  command,  to  ensure  that;  ( 1 ) 
an  integrated  baseline  review  on  the  last  two  releases  of  the  first  increment  is  conducted; 
(2)  compliance  against  the  32  accepted  industry  earned  value  management  (EVM) 
practices  is  verified:  and  (3)  a  plan  to  have  an  independent  organization  perform 
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surveillance  of  the  program’s  EVM  system  is  developed  and  implemented.  (Page 
52/GAO  Draft  Report) 

POD  RESPONSE:  Concur.  The  Department  appreciates  the  importance  of  Earned 
Value  Management  (EVM)  as  an  effective  management  tool  that  promotes  efficient 
management  of  programs  such  as  Navy  ERP.  Navy  ERP  has  been  implementing 
“Program-Level”  EVM  since  2006.  During  that  time,  the  Navy  ERP  Program  Office  has 
received  detailed  guidance  and  advice  from  various  government  and  industry  groups  since 
Program-Level  EVM  is  rare  and  requires  tailoring  to  achieve  compliance  with  best 
practices. 

A  key  pan  of  this  implementation  is  the  Integrated  Baseline  Review  (IBR),  which  has 
been  scheduled  for  Navy  ERP  Release  1 .1  in  August  2008.  Although  this  IBR  will  focus 
on  Release  1 .1 ,  it  will  also  include  the  remaining  Release  1.0  deployments.  The  IBR  will 
focus  on  the  health  of  the  baseline,  assessing  schedule  and  associated  resourcing.  Navy 
ERP  will  also  conduct  IBR  on  future  releases. 

An  IBR  was  not  conducted  on  Release  1 .0  because  once  the  rebaselining  was  approved  in 
December  2006,  development  was  essentially  complete  and  the  release  was  in  the  latter 
stages  of  testing.  However,  a  Schedule  Risk  Assessment  (SRA)  of  Release  1 .0  was 
conducted  in  March  2007.  The  main  finding  was  that  the  Integrated  Master  Schedule 
(IMS)  was  high  risk  for  achieving  Go-Live  due  to  the  lack  of  integration  of  the  Navy  ERP 
schedule  with  the  schedule  of  Naval  Air  System  Command  (NAVAIR),  the  customer. 
Navy  ERP  took  corrective  measures  to  mitigate  this  risk  by  developing  a  cut-over 
schedule  that  integrated  with  NAVAIR’s  schedule.  Navy  ERP  has  made  it  a  requirement 
to  incorporate  customer  schedules  in  all  future  releases.  The  corrective  measures 
successfully  mitigated  the  risk  associated  with  1 .0  as  indicated  by  the  Program  achieving 
Go-Live  as  scheduled. 

Navy  ERP  follows  the  direction  of  the  Assistant  Secretary  of  the  Navy  for  Research. 
Development  and  Acquisition  (ASN(RDA))  to  work  with  the  Navy  Center  for  Earned 
Value  Management  (CEVM)  to  validate  that  management  processes  meet  the  intent  of  the 
32  guidelines  described  in  the  American  National  Standards  Institute/Electronic 
Industries  Alliance  (ANSI/EIA)  standard  ANSI/EIA-748.  CEVM  will  also  perform 
ongoing  surveillance  of  Navy  ERP  processes  and  EVM  data,  addressing  the  last  element 
of  GAO’s  recommendation.  Additionally,  as  of  August  2008,  Navy  ERP  is  complying 
with  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (AT&L) 
memo  issued  July  11,  2007  on  the  implementation  of  the  Central  Repository  System. 

This  memo  requires  that  all  Acquisition  Category  I  programs  submit  monthly  cost  and 
schedule  performance  reports  via  the  Defense  Cost  and  Resource  Center  (DCARC) 

Earned  Value  Repository. 

RECOMMENDATION  3:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy,  through  the  appropriate  chain  of  command,  to  ensure  that  the 
schedule:  (1)  includes  the  logical  sequencing  of  all  activities;  (2)  reflects  whether  all 
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required  resources  will  be  available  when  needed;  (3)  defines  a  critical  path  that 
integrates  all  three  releases;  (4)  allocates  reserve  for  the  high-risk  activities  on  the  entire 
program’s  critical  path;  and  (5)  incorporates  the  results  of  a  schedule  risk  analysis  for  all 
three  releases,  and  recalculates  program  cost  and  schedule  variances  to  more  accurately 
determine  a  most  likely  cost  and  schedule  overrun.  (Page  52/GAO  Draft  Report) 

POD  RESPONSE:  Partially  Concur.  The  Department’s  EVM  policy  recognizes  that  the 
preparation  of  an  Integrated  Master  Schedule  (IMS)  is  a  best  practice.  The  Department 
concurs  with  the  following  elements  of  the  recommendation  and  the  Navy  ERP  Program 
Office  will  ensure  that  the  schedule: 

•  Includes  the  logical  sequencing  of  all  activities:  Navy  ERP  has  implemented  a 
logically  sequenced  schedule  for  Release  1.1  of  all  activities  based  on  experience 
gained  from  Navy  ERP  Release  1 .0.  Improvements  have  been  made  and  the 
schedule  is  continuously  monitored  through  weekly  schedule  metrics. 

•  Reflects  whether  all  required  resources  will  be  available  when  needed:  The  Navy 
ERP  IMS  has  a  resource-loaded,  time-phased  schedule  for  each  release  that 
identifies  resource  allocation  and  is  aligned  with  the  Program's  approved  staffing 
plan. 

•  Allocates  reserve  for  the  high-risk  activities  on  the  entire  program’s  critical  path: 
Navy  ERP  has  actively  begun  refining  the  risk  identification  process  to 
incorporate  risk  and  potential  impact  to  the  program  schedule  utilizing  schedule 
inputs.  As  this  process  matures,  the  result  will  be  used  to  assist  in  identify 
adequate  resources  required  in  managing  integrated  program  activities  across  all 
three  releases. 

•  Incorporates  the  results  of  a  schedule  risk  analysis  (SRA)  for  all  releases:  The 
Results  of  the  SRA  for  Release  1 . 1  will  be  compared  to  the  results  of  the  SRA  for 
Release  1 .0  and  incorporated  into  the  variance  analysis  for  cost  and  schedule  to 
avoid  any  potential  overruns.  Navy  ERP  is  implementing  an  estimate  at 
completion  (EAC)  process  for  quarterly  updates  beginning  in  Fiscal  Year  2009. 

The  Department  partially  concurs  with  the  third  element  of  the  recommendation,  that  the 
schedule  defines  a  critical  path  that  integrates  all  three  releases.  Navy  ERP  utilizes  a 
critical  path  for  each  release.  This  approach  recognizes  the  size  and  complexity  of  the 
Navy's  implementation  releases  and  still  allows  the  Navy  to  remain  focused  on  the  true 
dependencies  the  individual  releases  have  with  one  another.  The  Program  Office  will 
review  this  recommendation  with  Navy  leadership  to  determine  feasibility  for  Navy  ERP. 

RECOMMENDATION  4:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy,  through  the  appropriate  chain  of  command,  to  ensure  that:  (1 ) 
the  plans  for  mitigating  the  risks  associated  with  converting  data  from  legacy  systems  to 
Navy  ERP  and  positioning  the  commands  for  adopting  the  new  business  processes 
embedded  in  the  Navy  ERP  are  re-evaluated  in  light  of  the  recent  experience  with  Naval 
Air  Systems  Command  (NAVAIR)  and  adjust  accordingly;  (2)  the  status  and  results  of 
these  and  other  mitigation  plans’  implementation  are  periodically  reported  to  program 

Attachment 
Page  3  of  5 


Page  59 


GAO-08-896  DOD  Business  Systems  Modernization 


Appendix  II:  Comments  from  the  Department 
of  Defense 


oversight  and  approval  authorities;  (3)  these  authorities  ensure  that  those  entities 
responsible  for  implementing  these  strategies  are  held  accountable  for  doing  so;  and  (4) 
each  of  the  risks  discussed  in  this  report  are  included  in  the  program’s  inventory  of  active 
risks,  and  managed  accordingly.  (Page  53/GAO  Draft  Report) 

POD  RESPONSE:  Concur.  Navy  ERP  incorporated  lessons  learned  from  NAVAIR  into 
the  Navy  ERP  Program  Command  Implementation  Guidance.  The  guidance,  a  copy  of 
which  is  attached  for  reference,  incorporates  key  lessons  learned  from  prior  deployments 
and  provides  a  detailed  overview  of  the  implementation  process  and  key  information 
required  in  structuring  a  Command’s  implementation  team  and  activities  needed  for 
success.  The  Navy  ERP  Program  Command  Implementation  Guidance  also  identifies 
critical  success  factors  based  on  extensive  commercial  and  Navy  ERP  implementation 
experience  and  provides  a  time-phased  checklist  to  help  focus  a  Command’s  resources  on 
the  right  actions,  at  the  right  time.  It  has  been  reviewed  by  the  Navy  System  Command 
Commanders  and  is  being  used  by  the  Navy  Supply  System  Command  (NAVSUP)  in 
preparation  for  the  next  deployment  of  Navy  ERP  Release  1 .0. 

The  Readiness  Assessment  checklist  in  the  Navy  ERP  Program  Command 
Implementation  Guidance  helps  to  identify  risks,  and  through  the  Navy  ERP  Risk 
Program  process,  elevate  the  risk  appropriately  to  the  right  level  of  management  to 
facilitate  mitigation  planning.  In  addition  to  those  risks  identified  by  the  program.  Navy 
ERP  will  add  the  risks  identified  by  GAO  to  its  risk  inventory.  Risks  and  mitigation 
plans  are  formally  captured  by  the  Navy  ERP  Risk  Management  Board  in  its  Risk  Radar 
tracking  tool. 

Furthermore,  in  light  of  the  experience  learned  with  deployment  to  NAVAIR,  Navy  ERP 
has  completed  a  full  review  and  reevaluation  of  all  active  risks  and  corresponding 
mitigation  plans.  This  review  included  two  areas  noted  by  GAO  in  the  report:  (1) 
conversion  of  data  from  legacy  systems  to  Navy  ERP  and  (2)  transformation  change 
management  to  position  Commands  to  adopt  Navy  ERP. 

Since  Navy’s  ERP’s  deployment  of  Release  1 .0  to  NAVAIR,  the  enterprise  governance 
structure  that  reviews  program  risks  has  evolved.  Governance  now  includes  the  Navy 
Component  Acquisition  Executive’s  review  process,  the  Navy  ERP  Senior  Integration 
Board,  and  Enterprise  Risk  Assessment  Methodology  (ERAM)  assessments.  The  Navy 
will  work  through  these  governance  bodies  and  with  other  appropriate  oversight  and 
approval  authorities  to  determine  additional  reporting  mechanisms  for  risks  with 
associated  mitigation  plans,  to  ensure  commitment  and  accountability  across  the  Navy 
Enterprise.  Direction  received  from  these  authorities  with  respect  to  risk  and  risk 
management  will  be  incorporated  into  the  Navy  ERP  Risk  Management  Program. 

Through  the  Navy  and  Office  of  the  Secretary  of  Defense  (OSD)  governance  bodies, 
including  the  Milestone  Decision  Authority  (MDA),  Deputy  Under  Secretary  of  Defense 
(Business  Transformation),  as  well  as  Navy  Enterprise  Senior  Integration  Board  (NES1B), 
the  Navy  ERP  Program  Manager  is  held  accountable  for  overall  program  performance, 
including  implementing  risk  mitigation  strategies.  This  aligns  to  Department  of  Defense 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday,  GAO  posts 
newly  released  reports,  testimony,  and  correspondence  on  its  Web  site.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products  every  afternoon,  go 
to  www.gao.gov  and  select  “E-mail  Updates.” 
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The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each. 

A  check  or  money  order  should  be  made  out  to  the  Superintendent  of 
Documents.  GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or 
more  copies  mailed  to  a  single  address  are  discounted  25  percent.  Orders 
should  be  sent  to: 

U.S.  Government  Accountability  Office 

441  G  Street  NW,  Room  LM 

Washington,  DC  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 
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